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DEVELOPING A NATIONAL OUTCOME REPORTING SYSTEM 
FOR THE ADULT EDUCATION PROGRAM 

CHAPTER 1 

Starting Point: Background and Overview 

Like all publicly-funded service and education programs in recent years, the adult education 
program has faced increasing pressure, at both the state and national levels, to document to 
policymakers, legislators and the general public concrete outcomes resulting from program 
participation. Responding to this pressure, the Office of Vocational and Adult Education (OVAE) 
held a meeting of 12 state directors of adult education in January 1995 to discuss improving the data 
currently collected for program accountability. This meeting was followed by a policy forum held in 
the fall of 1995, where all state directors further discussed this issue. The state directors 
recommended to the U.S. Department of Education that efforts begin to develop an accountability 
system that was compatible with similar efforts of other agencies, and that would provide 
meaningful measures for policy and programmatic use at the Federal, state and local levels. 

To address this request, the U.S. Department of Education convened a national meeting of 
state directors of adult education in Columbia, Maryland, in March 1996. At that time, the state 
directors identified outcome measures to assess the impact of the adult education program on 
participants. At the conclusion of the meeting, the state directors recognized that additional work 
and development was necessary before a new accountability system could be implemented. In 
particular, there was concern that the efforts of adult education in measuring performance be 
consistent with the efforts of other programs and agencies. State directors also recognized that they 
needed further information and guidance from these other efforts. To promote continuation of their 
work, they adopted the following resolution: 

We recommend a collaboratively funded and managed project to analyze and synthesize 
accountability systems that have been developed nationally and in separate states that 
incorporate adult education outputs and outcomes. The project will continue the next 
steps of work begun here by state directors to draft protocols, determine how data 
would be collected and how reliability could be optimized. The project will involve state 
directors of adult education and other stakeholders in setting project policy and in 
project operations. 

In response to the resolution, OVAE created the National Outcome Reporting System 
Project. The goals of this project are to develop, test and implement a national reporting system of 
outcome measures that document the performance of students in the Federally funded adult 
education program. The project will begin in March 1997 when the state directors again meet to 
make initial decisions on the goals of the reporting system, measures to include and methodologies 
to pursue. 

This document has been developed to assist adult education stakeholders in developing a 
national outcome reporting system. It contains a review of existing and planned accountability 
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systems that include outcome measures relevant to adult education students. It also discusses many 
of the major issues that must be resolved in establishing an outcome-based reporting system. These 
issues draw from the experiences of other agencies and projects that have sought to establish similar 
systems. This report also summarizes methodological issues pertinent to the collection of the 
proposed measures and provides hypothetical examples of reporting systems for consideration. 

Building a Reporting System: A Starting Point 

The National Reporting Project will build upon initial work and decisions that have already 
been made. State directors have decided upon categories of measures and additional assumptions 
about the purpose and focus of the system through prior state director meetings and through other 
OVAE-sponsored projects on this topic. While these decisions are subject to review and may 
change, they serve as the starting point for the project and are summarized briefly below. 

Measures 

Depending upon their purpose, reporting systems often include several different types of 
measures to allow audiences to develop a picture of the program’s operation and performance. 
Measures of participant characteristics describe who is in the program, while measures of process 
reflect the activities of the program, such as the services it offers. System output measures indicate 
what the program produces, such as the number of instructional hours in adult education. All of 
these measures are important to documenting and understanding how a program performs. 

However, the development of the reporting system at this time will focus on identifying outcome 
measures. Outcomes can be defined as participant behavior, activities, opinions or learning that 
result from program participation and reflect progress toward achieving the purposes of the adult 
education program. While other measures will be included in a national adult education reporting 
system, the focus of development at this time will be upon identifying outcome measures. 

At the March 1996 meeting, the state directors developed the following seven categories of 
outcome measures for adult education that could be used to demonstrate the program's effectiveness 
to a broad audience. 

• Economic impact - measures that reflect a change in a participant’s employment or 
economic status, such as obtaining a job or going off public assistance. 

• Credentials - measures that reflect attainment of a diploma, skill certificate or other 
formal acknowledgment of completion of a program of study or training. 

• Learning gains - measures that demonstrate that the participant acquired reading, 
writing, functional or employment-related skills, numeracy, or English-speaking and 
listening skills. 

• Family impact - measures that reflect improved literacy-related activities with family 
members, such as reading to children, or greater involvement in children’s school-related 
activities. 

10 
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• Further education and training - measures of a participant’s continuation in an 
education or training program after leaving the adult education program. 

• Community impact - measures of improved community or civic involvement, such as 
voting, achieving citizenship or increased community activism. 

• Customer satisfaction - measures demonstrating the degree of a participant’s 
satisfaction with adult education instruction and whether services helped participants 
achieve their goals. 

The reporting system also is envisioned as a national system, meaning that all states will 
need to report a uniform set of quantitative measures using an individual student record system at the 
local level. The Department of Education will aggregate these measures to develop a national data 
base for reporting on the adult education program. However, the adoption of a national set of 
measures does not preclude other measures that individual states may collect. States can continue to 
collect additional measures for their own reporting purposes, as they do now. 

Although measures are an essential component of a reporting system, many of the basic 
issues inherent to the successful operation of such systems were not addressed at the meeting. For 
example, not discussed explicitly were the policy goals the measures reflected, the purposes for 
which they were to be collected, or the methods states were to employ to collect them. Such issues 
are typically decided prior to the development of measures. 

Purposes of the System 

While reporting and accountability systems can serve many purposes, they all are designed 
to assess progress of a program in meeting policy goals at both the state and Federal levels. 
Performance measures also can be a tool for program improvement by allowing states and local 
programs to identify areas of low performance. Accountability for program improvement has been a 
priority of OVAE and the states in their prior efforts in this area, such as development of quality 
indicators (OVAE, 1992). The reporting system to be developed should, therefore, meet these 
purposes: 

• To improve the public accountability of the adult education program by documenting its 
ability to meet its policy and programmatic goals at the Federal level; 

• To assist states to assess their progress in meeting their goals for adult education; and 

• To enable the Federal government, states and local programs to promote continuous 
improvement of the quality of instruction and services. 

One of the first tasks of the National Reporting Project will be to clarify the system's purpose and to 
define it more precisely. 
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Table 1 summarizes the measures, definitions and current conceptions about the reporting 
system that we call the Columbia Framework, after the meeting location where many of these issues 
were discussed. Although the Columbia Framework serves as the starting point for the project, it 
will continue to evolve as the project progresses. 



TABLE 1 



Columbia Framework for a National Outcome Reporting System for Adult Education 



Purposes of the System 



• To improve the public accountability of the adult education program by 
documenting its ability to meet its policy and programmatic goals at 
the Federal level. 



• To assist states to assess their progress in meeting their goals for adult 
education. 



• To enable the Federal government, states and local programs to 
promote continuous improvement of the quality of instruction and 
service. 

Measures 



• Economic Impact 

• Credentials 

• Learning Gains 

• Family Impact 



Further Education and Training 
Community Impact 
Customer Satisfaction 



Essential Characteristics 



National in scope — All states will report a uniform set of quantitative 
measures using the same methodology. However, states can continue to 
collect additional measures for their own purposes. 

Outcome-based — The system will measure learner outcomes — defined as 
learner behavior, activities, opinions or learning that results from program 
participation and reflects progress toward achieving the purposes of the 
adult education program. 

Individual student record system — Local programs will maintain an 
automated individual record system to collect and report measures. 



Overview of the Report 

The development of a national outcome reporting system for adult education is a complex 
activity that will take several years to implement. Fortunately, this project is not the first attempt of 
adult educators to improve their measures and system of accountability. Such efforts have been 
underway at the state and Federal level for several years. Since 1991, OVAE has supported state 
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efforts to improve accountability through the development of an evaluation framework, through the 
support of the development of quality indicators for adult education, and through the Technical 
Assistance Project for State Accountability and Assessment. The National Institute for Literacy 
(NIFL) also has recently completed a project to develop accountability systems in which adult 
education would play a key role. Other Federal agencies — notably, the Department of Labor and the 
National Governors’ Association (NGA) — have supported major efforts at developing accountability 
systems that have relevance to adult education programs. These projects, along with efforts of 
individual states, have produced a rich foundation of knowledge and experience from which to draw 
in the development of a national reporting system for adult education. 

This report summarizes these prior efforts and draws from their experiences to provide 
guidance in resolving the key issues confronting the development of a national outcome-based 
reporting system. Chapter 2 reviews existing systems and projects that include measures of adult 
education and training outcomes. A brief review of each project summarizes its operation, measures 
and methodology, current status, and issues encountered in its implementation. 

In Chapter 3, we present some of the major issues that need to be considered immediately in 
developing an accountability system, including goals of the system, audience, measurement issues, 
performance standards and avoiding unintended effects of performance assessment. Our discussion 
in this chapter draws from experiences of the NGA and NIFL accountability system projects. 

Chapter 4 confronts the issue of how to measure outcomes. We review potential measures of 
the reporting system by describing how they could be defined and compare different methodologies 
that could be employed to collect data on the measures. Methodologies are assessed by their ability 
to produce valid and reliable data, as well as cost and difficulty of implementation. Chapter 5 
concludes the report by describing four hypothetical examples of outcome-based reporting systems 
for the adult education program and assessing the advantages and difficulties of implementing each 
system. These sample systems are offered to promote thought and discussion as the national 
outcome reporting system develops. 
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CHAPTER 2 

Where We Are Now: Measures of Adult Education Student Outcomes 

There are currently several on-going efforts to develop, to implement or to collect outcome 
measures assessing the impact of adult participation in education and training programs. While 
some of these accountability systems have been operating for several years, many are new projects 
that are still in the planning phase and are yet to be implemented. Many of these systems provide 
valuable lessons and experiences that can inform the efforts of adult educators as they develop a new 
national reporting system. 

In this chapter, we review existing, or planned, national and state accountability systems that 
include measures of adult education and training outcomes in the areas identified by the Columbia 
Framework. Included in this review are reporting systems, demonstration projects and pilot 
programs that measure student and other customer outcomes. As shown in Table 2, the systems and 
projects reviewed here include systems currently in operation, planned reporting systems, 
demonstration projects, and state initiatives. 1 

TABLE 2 

Accountability Systems and Projects Reviewed 



Existing Systems^ *’ 


• 


Division of Adult Education and Literacy's Annual Statistical Report 


• 


Job Training Partnership Act Performance Standards System 


• 


National Center for Family Literacy Reporting System 


Planned Reporting System 


• 


One Stop Career Center System 


Demonstration Projects 


• 


Workforce Performance Management System of the National Governors' 
Association 


• 


Performance Measurement, Reporting and Improvement System Initiative of 
the National Institute for Literacy 


State Initiatives 


• 


California Performance-Based Accountability Project 


• 


Florida Education and Training Program 


• 


Oregon Workforce Programs Performance Measures 


• 


Texas Performance Measures Assessment System 



' The other Federally funded adult education and training programs (i.e., the now-defunct Job Opportunities 

and Basic Skills (JOBS) program, and Even Start) are not considered here since they do not have national 
participant outcome reporting requirements. The vocational education and school-to-work programs are not 
included as they primarily do not target adults. 
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For each system, we present: 

• A brief summary of its operation and activities; 

• The outcome measures collected or planned, organized by the Columbia Framework; 

• The methodology for collecting the measures; 

• The current status of the project; and 

• Issues encountered in implementing the project or collecting the measures. 

For projects that include multiple agencies, we also list all agencies involved in planning or 
implementing the system. 

These descriptions are intended to inform discussions about the specific measures that might 
be considered for a national outcome reporting system for adult education, possible methodologies 
for collecting the measure, and issues that need to be addressed when developing an outcome 
reporting system. They are not intended to provide a comprehensive understanding of all aspects of 
the respective projects. The content of the review is based upon reports and documents from 
national program offices and materials from state projects. Additionally, telephone interviews were 
conducted with staff from national and state programs. A list of sources and contacts is provided at 
the end of this document. 



Existing Reporting Systems 

Three existing reporting systems offer information that should be considered in developing a 
national outcome reporting system for adult education. Two of these systems operate through a 
Federal program structure — the annual statistical reporting system for the Adult Education Program, 
and the Job Training Partnership Act Performance Standards System. The third system described is 
operated through the National Center for Family Literacy. 

Annual Statistical Report for the State-Administered 
Adult Education Program 

The current Federal reporting requirements for the Adult Education Program are embodied in 
six tables of demographic and outcome data on students attending adult basic education and English- 
as-second language classes, and in two additional tables that collect information on provider 
agencies and state and local staff. Reporting requirements also include a financial status report and a 
narrative describing state progress toward meeting goals of the state plan, other significant activities 
and cost per student. All states are required to submit this information annually to the Division of 
Adult Education and Literacy (DAEL). 
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Outcome Measures 

Table 3 and Table 6 of the Annual Statistical Report include all of the categories of student 
outcome measures of the Columbia Framework, except customer satisfaction. 

• Learning gains. Upon intake, students are identified as either ABE or ESL students, and 
then are classified into one of four educational functioning levels. For ABE, the levels 
are beginning literacy, beginning, intermediate or adult secondary (GED); and, for ESL, 
the levels are beginning literacy, beginning, intermediate or advanced. The number of 
students advancing from each level during the year is the measure of learning gains. 

• Economic impact. The number of students that gained employment, retained 
employment or obtained job advancement, and that were removed from public 
assistance. 

• Credentials. The number of students who obtained an adult high school diploma or the 
GED. 

• Family impact. Family literacy programs report the number of students that read more 
to their children and have greater involvement in their children’s schooling. 

• Further education and training. The number of students who entered another education 
or training program. 

• Community impact. The number of students that registered to vote or voted for the first 
time and the number that received U.S. citizenship. 

Methodology for Collecting Measures 

Local programs report the necessary data to their state education agency (SEA) for the 
Annual Statistical Report at least annually. The SEA then aggregates the data from all local 
programs for annual reporting to the U.S. Department of Education. Local programs report on all 
students who attend 12 hours or more during the program year. Each state is responsible for 
implementing its own methodology for collecting the measures. While there are common 
definitions of the measures, there is no standard methodology for collecting the data. 

For the measure of educational advancement, local programs are responsible for initially 
classifying each student into one of the educational functioning levels and then assessing the 
student’s progress prior to exit from the program. Programs use their own methods for assessing 
students and many states require local programs to use standardized tests, typically CASAS and 
TABE for ABE students, and the BEST, CELSA or CASAS-ESL for ESL students. 

Local programs collect the other outcome measures using a variety of methods that vary 
within and across states. There currently exists no complete or systematic accounting of 
methodologies used by states and local programs for obtaining these measures. 
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Current Status 

The Annual Statistical Report is the current reporting system for all states and programs 
receiving funds from the Basic State Grant authorized by the Adult Education Act. 

Issues 

Three major issues in the collection of data for the Annual Statistical Report have been the 
lack of a clear definitions for educational functioning level, the lack of sufficient resources at the 
local level for collecting follow-up outcome data, and incomplete reporting of these follow-up data. 
To address the problems with functioning-level definitions, DAEL developed new.definitions for the 
1996 program year that describe the functional skills, reading and writing skills, computational 
abilities (for ABE) and speaking and listening skills (for ESL) of students entering at each level. 

Tied to each level are suggested test benchmarks that states optionally may use to assess students. 
States are free to use their own definitions and benchmarks as long as they can map them to the 
Federal definitions. 

The minimal resources for data collection of most local programs has resulted in what is 
likely an incomplete accounting of the number of students achieving the economic, community and 
further education outcomes. The degree of coverage and validity of these data are unknown, as the 
1996 program year is the first year states have had to report the family outcome data. 

Job Training Partnership Act Performance Standards System 

The Job Training Partnership Act (JTPA) requires the establishment of performance 
standards that measure “. . .the increased employment and earnings of participants and the reduction 
in welfare dependency (Section 106)”, that reflect the volume and quality of placements and that 
demonstrate cost-effectiveness. 2 Specific objectives of the JTPA performance standards, which were 
revised in 1990, include: 

• Holding service delivery areas (SDAs) accountable for program outcomes. 

• Encouraging the achievement of high-quality employment outcomes, including increases 
in the long-term employability of participants. 

• Creating an incentive for effective management by local program administrators. 

• Fostering acceptance of JTPA by businesses and the public by demonstrating program 
accomplishments. 



2 The Department of Labor also wanted to use these performance standards to help restore credibility to Federal 
employment programs that had been eroded by JTPA’s predecessor program, the Comprehensive Employment and 
Training Act. 
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Among the basic principles followed by DOL in establishing the JTPA outcome-based 
performance system are that standards should: 

• Measure the achievement of key program objectives, 

• Be easily measurable for individual SDAs, 

• Be comparable among SDAs, 

• Avoid unintended consequences, 

• Be understandable within and outside of the JTPA system, 

• Be limited in number, and to avoid confusing SDAs and to minimize the influence of 
individual standards, and 

• Provide information on performance to SDAs quickly so that feedback is not 
unnecessarily delayed. 

Outcome Measures 

JTPA has established six performance measures — four for adult programs and two for 
youth programs. 

• Economic impact. All of the measures of the adult programs, and one youth outcome 
measure, assess employment. 

1 . Adult follow-up employment rate — the percent of participants employed 1 3 
weeks after leaving the program. 

2. Adult weekly earnings at follow-up — average weekly earnings of those 
employed 13 weeks after leaving the program. 

3. Welfare follow-up employment rate — the percent of welfare recipients at 
application who are employed 13 weeks after leaving the program. 

4. Welfare weekly earnings at follow-up — average weekly earnings of welfare 
recipients who are employed 13 weeks after leaving the program. 

5. Youth entered employment rate — percent of youth employed when leaving the 
program (youth who remain in or return to school are not counted in the 
denominator unless they also are employed). 

• Learning gains. Although the Department of Labor has considered adding one for 
several years, there are no performance standards that assess adult learning of 
employment-related skills. For assessing the progress of youth (participants aged 16 to 
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21) programs can report the youth employability enhancement rate, defined as the 
percent youth attaining any of the following three outcomes deemed to affect 
employability: 

1. Attained competency in two of three skill areas: basic education skills, pre- 
employment/work maturity skills, and job-specific skills. 

2. Remained in full-time school, including retention for 120 days, making satisfactory 
progress, and attaining competency in one of the three skill areas. 

3. Returned to school full-time, including retention for 120 days, making satisfactory 
progress, and attaining competency in one of the three skill areas. 

• Credentials. Number of youth completing a major level of education (e.g., graduated 
from high school or obtaining a GED) also can be counted in computing the youth 
employability enhancement rate. 

• Further education and training. Another acceptable measure of youth employability 
enhancement is the number of youth who entered non-JTPA training, and stayed for at 
least 90 calendar days or 200 hours. Certification of occupational skill attainment can 
substitute for the retention requirement. 

DOL allows localities to define the specific skills to be measured as learning gains within the 
youth employability enhancement rate. Definitions must address basic skill and occupational 
competencies, satisfactory progress and locally-recognized competencies, as defined below: 

• Basic skills competencies. These skills are usually established by consulting with local 
school boards and include academic areas such as reading, writing, mathematics, or 
language arts. Benchmarks may include improvement in test scores or in grade point 
average. 

• Occupational skill competency. Occupational skills are specific skills required for a 
certain job. States and localities define these skills according to their labor market needs. 
Examples of these skills are familiarity with equipment or speed using equipment. 

• Satisfactory progress. To make satisfactory progress, JTPA participants must improve 
one step from where they were at enrollment either in the area of education, 
employment, or employment readiness. For instance, a participant may compete the next 
major education level by obtaining either a GED, Associate degree, or Bachelor degree. 
Alternatively, a participant may acquire an occupational skill competency. 

• Locally recognized competency. Localities may establish their own basic and 
occupational competencies that help participants obtain jobs and behave professionally 
on the job. Examples of these skills are resume writing, discipline, punctuality, and 
proper dress. 
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Methodology for Collecting Measures 

Local providers use the JTPA management information system to record immediate training 
outcomes and to compute employment rates. For the follow-up measures, most providers use third 
party contractors to contact former participants by mail and by telephone to obtain the needed 
information. Some providers use unemployment insurance records to locate former participants. 

The most recent JTPA amendments encourage measuring participant employment retention at least 
six months after leaving the program, although the current outcomes are based upon three months 
after leaving the program. DOL believes that such a change, while requiring minimal changes in 
data collection methodologies, would reduce the response rates from former program participants. 

At present, 80 percent of providers reach response rates above the required 70 percent, although 
samples sizes are small. 

Current Status 

Both DOL and states have responsibilities in the JTPA performance system. DOL is 
responsible for: (1) selecting the outcome measures and national performance standards; (2) 
developing procedures for states to adjust standards so they reflect local demographics and economic 
conditions; and (3) determining the performance levels considered to be acceptable and those 
considered to represent failure. 

States are responsible for: (1) setting standards for each SDA after considering participant 
demographics and local economic conditions; (2) defining the point of failing, meeting, and 
exceeding individual standards; (3) developing procedures for distributing incentive funds to SDAs 
that exceed performance standards; and (4) developing policies for providing SDAs with technical 
assistance to improve their performance. 

JTPA performance standards are used in a number of ways. First, five percent of JTPA 
funds are reserved for incentive awards to SDAs based on their performance according to the 
outcome measures. Second, SDAs with problems, as identified by low performance on the outcome 
measures, receive technical assistance. Beginning with the 1994 program year, the Department of 
Labor defined overall failure as failing three of the six adult and youth standards, or both of the 
youth standards. 

Issues 

One of the major problems experienced by JTPA programs resulting from the use of 
performance standards is the unintended consequences the standards have on program service 
delivery and participant enrollment. Studies of the impact of standards have found that, when the 
standards are set too high, local programs become overly concerned about meeting the standards and 
may design programs to meet the standards, rather than serve clients. For example, a high 
employment rate standard can cause programs to design employment activities that can place many 
participants quickly, such as job search, and avoid more intensive job-training programs that often 
take more time. In addition, programs may be motivated to allow only more job-ready applicants 
into the program, shutting out participants with lower skills who are harder to place and who require 
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more intensive training. This well-known “creaming” effect was one reason DOL abandoned one of 
JTPA’s first performance standards, cost per entered employment. 

A second difficulty with performance standards is that collecting the participant follow-up 
data has proven to be very costly. These surveys also have low sample sizes and are subject to 
nonresponse bias, which cannot be completely controlled through data analysis methods. Finally, 
since the youth skill attainment measures are locally defined, they are not measured consistently 
across service delivery areas, making it difficult or impossible to aggregate them to a state or a 
national level. In addition local control has made the outcomes based upon skill attainment easy to 
manipulate, such as by choosing easy to meet criteria, to do well on the standard. 

National Center for Family Literacy Reporting System 

The National Center for Family Literacy (NCFL), founded in 1989, funds family literacy 
programs to providers throughout the country. Programs funded by NCFL have the following four 
components: early childhood education, based upon developmentally appropriate preschool 
curriculum; adult education, including GED preparation and ESL; parenting education, which 
includes parent support groups; and parent-child interaction time, where parents play and learn with 
their children. As part of its evaluation efforts to develop data for program improvement and to 
demonstrate program effectiveness, NCFL requires all of its grantees to provide data annually on 
families attending their programs. Grantees collect a wide range of demographic and outcome data 
using 10 forms developed by NCFL. 

NCFL’s reporting system includes outcome measures for parents and children, measures of 
parenting skills and parent self-esteem measures. NCFL encourages grantees that want more in- 
depth information in particular subject areas to use additional measures beyond those on NCFL 
forms. For example, programs wanting more information on children’s cognitive abilities could 
assess children using one of several available standardized tests. NCFL has developed a summary of 
adult and child assessment instruments to assist local grantees. 

Outcome Measures 

The measures of adult student outcomes fall into the following categories of the Columbia 
Framework. NCFL collects measures of child outcomes and child development, along with the adult 
outcome measures. 

• Learning gains. Pre- and post-test scores on the TABE are reported for ABE students. 
Scores on GED tests are reported for students taking these tests. Grantees are to provide 
tests scores for ESL students using whatever standardized test they wish. 

• Economic impact. Whether the student obtained or improved employment. 

• Credentials. Whether the student attained a GED. 

• Family impact. The parent survey form measures family impact outcomes by having 
parents indicate the number of times in the past week they performed each of the 
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following activities: read to their children, talked with their children about school, took 
their children to the library, helped their children with homework, talked with their 
children’s teachers and attended or volunteered to help with activities at their children’s 
school. 

• Further education and training. Whether the student entered community college or 
university, or a vocational or employment training program. 

Methodology for Collecting Measures 

Grantees return the individual record forms to NCFL, which then creates a national data 
base. Forms are returned three times a year: October, February and at the end of the program year 
(i.e., July 1). 

An exit form, completed when a family member leaves the program, records the employment 
and further education and training outcomes. For the family impact measures, the parent’s initial 
parent survey form, which includes the measures, serves as a baseline or pretest score, and measures 
on forms sent later serve as post-test scores. The TABE is administered within 2 weeks of intake 
and posttests are given after every 100 hours of instruction. Separate forms report GED tests and 
GED attainment. 

Current Status 

The NCFL Reporting System began in 1991 and now contains data from over 5,000 families 
in 75 family literacy programs. The forms were revised for the 1996-1997 program year. 

Issues 

NCFL reports that programs experience difficulties in completing reports due to lack of 
resources available for reporting activities. In an effort to improve response rates, NCFL has 
provided additional funds to grantees for reporting, but these funds have not resulted in substantial 
improvements in reporting. In addition, there are relatively few TABE post-test scores 



Planned Reporting Systems 

A national reporting system for DOL’s One Stop Career System is currently under 
development. DOL began the One Stop System with grants to six states in 1 994. 

One Stop Career Center System 

The One Stop Career Center System is an initiative by DOL to develop an integrated 
employment service delivery system for job-seekers and employers. The Centers are to offer 
integrated employment, training and education services to permit seamless service delivery to Center 
users. The Centers also are required to have an integrated governance structure. 
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Besides the requirement for integrated services, DOL established three additional principles 
for One Stop Centers: 

• Universality - All populations are to have access to the Center’s assessment services, 
self-help information for career exploration, information on performance of local training 
providers and labor market information. 

• Customer Choice - Centers are to promote customer choice by offering information to 
help users make informed choices on providers and training opportunities. 

• Performance Driven/Outcome-based — Each Center must specify the outcomes it plans 
to achieve, have a system for measuring these outcomes and consequences for failing to 
achieve the outcomes. Customer satisfaction must be included as a measure of success. 

Six states — Connecticut, Iowa, Maryland, Massachusetts, Texas and Wisconsin — received 
grants to develop One Stop Centers in 1994, and 10 additional states — Arizona, Illinois, Indiana, 
Kentucky, Louisiana, Minnesota, Missouri, New Jersey, North Carolina and Ohio — received grants 
in 1995. In late 1996, 17 additional states received One Stop grants. The states have considerable 
latitude to develop delivery systems that meet state needs. The only requirement from the 
Department of Labor is that the Centers’ activities implement the four themes of integration, 
universality, customer choice and performance driven/outcome-based operation. 

Outcome Measures 

Each state is responsible for developing performance measures for its One Stop Centers that 
can be used to assess progress toward achieving the four principles. The grantee states developed 
the One Stop Performance Measures Committee, an ad hoc working group, to share information and 
to address common concerns in developing their measures. As of June 1996, 1 1 states had 
developed measures, at least in draft form. 

A DOL review identified four types of measures: process measures that measure activities 
the Centers perform; output measures that reflect the services and products produced; satisfaction 
with the centers, expressed by employers and job seekers; and outcome measures (KRA, 1996). The 
outcomes measures reflect the goals of the Centers to help job seekers obtain meaningful 
employment or education. The outcome and satisfaction measures have the greatest relevance to an 
outcome-based adult education reporting system. While these measures and their definitions vary 
somewhat across states, at least three states operating One Stop Centers have each of the following 
measures, organized by the Columbia Framework. 

• Learning gains. Percent of students who attain skills that are needed to succeed in the 
workplace. 

• Economic impact. Most outcome measures for One Stop Centers fall into this category. 
There are two type of economic measures: employment/eamings, and return on 
investment. The four employment/eamings measures are: 
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1 . The number who entered employment; 

2. Placement rate (i.e., number placed in jobs divided by total number of job seekers); 

3. Job retention after 90 days or longer; and 

4. Earnings, measured as the average starting wage of jobs obtained. One state 
measures earnings as the increase, or percentage change, in earnings compared to 
earnings at the job prior to entry into the One Stop Center. 



The return on investment measures assess whether society as a whole benefits from One 
Stop Center services. These measures include welfare reduction measures, or the 
number of clients who would have been on welfare, but instead obtained employment 
through One Stop centers; welfare savings, measured as the amount of money saved by 
state and local governments from clients who would have received welfare benefits, but 
instead obtained employment; and unemployment insurance (UI) savings, measured by 
comparing UI benefits paid to One Stop Center participants and UI benefits paid to the 
total claimant population. Two states also plan to measure total dollars saved by 
comparing total expenses spent on behalf of One Stop clients and the amount of taxes 
paid by Center clients who obtain jobs, with estimated amounts that would have been 
spent on welfare and unemployment benefits. 

• Credentials. The number of students who attain a GED, high school diploma or 
professional or occupational credential through program services. 

• Further education and training. The percentage of high school leavers who enter post- 
secondary education or training, a registered apprenticeship program or Job Corps. 

• Customer satisfaction. Satisfaction of job seekers and employers with the One Stop 
Centers’ services and training. 

Methodology for Collecting Measures 

Since each state defines and collects its own outcome measures, there are variations in 
definitions and methodologies across the One Stop states. However, as the One Stop Centers are not 
yet fully operational, few Centers have yet collected data. In general, individual student record 
systems operated by One Stop Centers will be the primary source of employment outcome data. 

Each Center will record the education or training services received and whether the client obtained a 
job. These data will allow computation of placement rates and number of clients who obtained 
employment. Centers will conduct follow-up surveys of a sample of clients to obtain earnings and 
job retention data. 



The planned return on investment measures will be computed, over a 3 to 10 year period, 
through models that combine actual program costs, estimates of taxes paid and benefits saved by 
welfare and UI agencies, adjusted for inflation. To compute these measures, data will be shared 
among welfare, UI and One, Stop Centers. Some states also will develop interagency data systems 
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for follow-up to determine whether clients obtain a GED and enter other employment and training 
programs. 

States plan several methods for measuring employer and job-seeker satisfaction with the 
Centers. These methods include exit surveys, follow-up mail or telephone surveys of samples of job 
seekers, focus groups and interviews at the Centers of small samples of job seekers. 

Current Status 

The One Stop Center grants are for three years and states are in varying stages of 
implementation — either in the planning phase or only recently begun service delivery. No 
performance data from any state are yet available. DOL began a new project in 1997, the Workforce 
Development Performance Measure Initiative, which will continue developing performance 
measures for the emerging workforce development system. The initiative will include input from 
the Departments of Education, Health and Human Services, and Housing and Urban Development, 
to develop a menu of measures with standard definitions for workforce programs. 

Issues 

Due to the recency of the One Stop Center initiative, DOL has not yet identified issues 
associated with implementing the outcome reporting system. 



Demonstration Projects 

Two national demonstration projects offer information that should be considered in 
developing a national outcome reporting system for adult education: the National Governors’ 
Association (NGA) Workforce Performance Management System, and the National Institute for 
Literacy’s (NIFL) Performance, Measurement, Reporting, and Improvement System (PMRIS). 

Workforce Performance Management System of the National Governors’ Association 

During the summer of 1994; the NGA, began working with six states to develop a framework 
for managing and measuring performance across workforce development programs. The six states 
participating in the project were: Illinois, Iowa, New York, North Carolina, Texas and Washington. 
In each state, interagency teams, including representatives from employment, training, education and 
welfare agencies, participated in the project and developed measures for a workforce performance 
measurement system. 

Table 3 presents the agencies participating in the development of workforce performance 
management systems in the six states. Table 4 summarizes measures used for the workforce 
performance management systems in each of the six states. We briefly discuss each project 
separately for all of the states, except for New York, since activities varied significantly across the 
states. 
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Table 3 



Programs Included in the State Workforce Development System 




Job Training Partnership Act, Title II 
Dislocated Workers 
Older Workers 

Job Opportunities and Basic 
Skills Training programs 
Employment Security 
(Wagner-Peyser) 

Vocational/T ech nical Education 

• Secondary 

• Postsecondary 

Higher Education 
Vocational Rehabilitation 
Corrections 
Adult Education 

Literacy 

State-funded 

Federally funded 

State Customized Training 

Trade Adjustment Assistance 

Food Stamp Employment and Training Program 

Veterans 

Americorps 



Source: Adapted from Trott and Baj, 1996 
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Table 4 



Outcome Measures Selected by the NGA Workforce 
Development System 



yy- t' Measures V ' r. Illinois ; iowa ;NewYork ■ 



Washington y 



Credentials 














• Attained certificate of initial mastery or 
equivalent 


• 






• 




• 


• Attained high school diploma 


• 


• 


• 


• 


• 


• 


• Attained High school equivalency 
certificate 


• 


• 


• 


• 


• 


• 


• Attained postsecondary degree or 
certificate 


• 


• 


• 


• 


• 


• 


• Attained a skills credential 


• 


• 


• 


• 




• 


• Attained a competency credential 


• 




• 


• 


• 




Leaminq Gains 














• Adult attainment of significant increase in 




• 




• 


• 


• 


educational competencies 














Further Education and Trainina 














• Returned to school 

• Entered postsecondary training 


• 


• 




• 


• 




Economic Impact 














• Entered unsubsidized employment 

• Entered with an earnings constraint 




• 

• 


• 


• 


• 


• 


• Entered subsidized employment 












• 


• Retained employment 


• 


• 


• 


• 


• 


• 


• Entered employment related to training 




• 


• 


• 


• 




• Wage at placement 




• 


• 


• 


• 


• 


• Earnings gains 


• 


• 


• 


• 


• 


• 


• Average earnings in a post-program 


• 


• 


• 








period 












• 


• Welfare reduction 


• 


• 


• 


• 




• 


• Unemployment insurance savings 


• 


• 


• 








Other 














• Participation equity rate 




• 


• 


• 




• 


• Attainment of successful outcomes by 
target population 


• 




• 


• 


• 





Source: Adapted from Trott and Baj, 1996 
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Illinois 



Illinois’ project identified six building blocks necessary to develop a comprehensive 
performance management system: (1) state policy goals and benchmarks, (2) policy outcome 
measures, (3) common data collection information management strategies, (4) performance 
standards for system-program-supplier, (5) supplier certification and performance consequences, and 
(6) state evaluation and continuous improvement systems. To guide development of this system, 
Illinois first adopted a vision statement for its policy goals and benchmarks during the NGA pilot 
project, which promotes: 

• Equity of access, services, and opportunities for all Illinoisians; 

• Attainment of foundation academic, workplace and occupational skills necessary for 
employment and further education and training; 

• Ongoing skills development to support continued employability; 

• Economic self-sufficiency through rising real incomes; 

• Increased competitive advantage for Illinois businesses in the global marketplace; and 

• High quality, efficient and accountable workforce development programs. 

Most of the work conducted during the NGA project involved identifying common measures 
and measurement strategies that would function as the outcome-based accountability portion of the 
system. Measures included participant outcomes relating to academic achievement, occupational 
competence and employment-related measures, as well as “social benefit” measures relating to 
welfare savings and unemployment insurance reductions. As a result of the progress made in 
identifying common measures and measurement strategies, a pilot test of the employment and 
earnings measures was conducted, drawing upon program completer data from the major state 
workforce agencies. Work on developing the workforce performance measurement system is 
continuing under Illinois’ One-Stop Career Center grant. 

Methodology for Collecting Measures 

Northern Illinois University’s Center for Governmental Studies is under contract with the 
Department of Commerce and Communication Affairs (DCCA) to assist with data collection and 
piloting the outcome measures. The Center negotiated the receipt of participant data with each of 
the major state workforce agencies to form a statewide participant data base with more than 1 
million records. As part of this effort, an unduplicated count of individuals participating in the 
Illinois workforce system is being calculated, co-enrollment and sequential enrollment among 
programs are being identified to address questions of system capacity and coordination, and a 
trial run of most performance measures is being done. Additionally, policy papers on various 
performance management strategies were prepared. A third year of participant data is being 
collected from each state workforce agency by the Center for Governmental Studies to identify 
performance trends. 
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Current Status 

The Illinois Common Performance Management System Project (ICPMS) is continuing in its 
efforts to develop and to implement a statewide outcome and program management accountability 
system that crosses all major workforce preparation programs, including adult education. In 1996, 
continuation of the project was identified as a state priority in the Illinois Workforce Preparation 
System and Action Plan, and also was endorsed by the Illinois Human Resource Investment Council 
(HRIC). Implementation of the system was identified as one of the three top priorities for 1997 by 
the HRIC’s Information, Accountability and Research Committee. 

Issues 

Four issues have been identified by staff responsible for the NGA project in Illinois. 

1 . An ICPMS Technical Team was recently convened to examine the performance 
measures identified at the policy level to determine: (1) whether data were currently 
collected to use the measures; (2) whether current data could be used to run a 
“comparable” measure from that suggested by the outcomes, if necessary; and (3) the 
cost or benefit of adding data elements to required reports of the various agencies. 
Modifications to some of the measures are anticipated. 

2. A subcommittee of the state ICPMS Policy Team is being convened to include 
representatives of local education and training agencies. The purpose of this 
subcommittee is to examine issues associated with “consumer reporting” of performance 
information to support informed customer choice. This is a priority activity under the 
One Stop Career Center Implementation Grant and is likely to be among the most 
controversial. 

3. The Illinois State Board of Education (ISBE) has expressed an interest in including 
results from the Illinois Goal Assessment Program, measuring academic achievement to 
the ICPMS. In addition, the Illinois General Assembly passed legislation last year, 
requiring the establishment of a Prairie State Achievement Examination and credential. 
The ICPMS measures need to be reviewed in light of this action. Similarly, the ISBE 
will be examining the relationship between the Prairie State Exam and adult education 
performance standards, to ensure consistency. There should not be different expectations 
of achievement in education for youth and adults. 

4. Various state agencies are examining their current “program-specific” outcome measures 
to determine whether ICPMS measures and measurement strategies can be substituted 
for current requirements. For example, when the state adult education community was 
developing its performance standards system, significant discussion occurred around 
adopting the GED attainment measures under ICPMS as part of their outcome measures. 
Other programs are examining the potential of Federal waivers. 
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Iowa 



Iowa’s performance management system is intended to include as many types of measures as 
appropriate for workforce development activities. The Iowa system is designed to promote shared 
accountability among programs within geographic regions of the state. To the extent possible, the 
goal is to minimize data collection burdens and to replace the existing data collection and 
measurement system with an automated system. 

Methodology for Collecting Measures 

The working committee suggests that analyses of the measures be conduced on a routine and 
ongoing basis and be reviewed in view of “best practices” for workforce development with the 
explicit purpose of improving both policy and practice. No other specification was given. 

Iowa is considering collecting data about customer value of services through four different 
methods: 

1 . Surveys, which can collect data that are objective and easy to quantify. 

2. Focus groups, which provide more detailed information about the needs and interests of 
program participants, although data from focus groups are typically more subjective than 
those from surveys. 

3. Anthropological observation of program participants as they receive program services, 
which can identify where services are being provided appropriately and where there are 
problems. 

4. Consultation with line staff who have the closest relationship with program participants 
and typically possess a good understanding of the aspects of components of programs 
that participants consider to be most valuable and effective. 

Current Status 

The project hopes to receive formal approval on the measures in Spring 1997. Assuming 
approval, the plans are to develop an integrated information system by the summer and then to pilot 
test a common intake prototype at three local sites. 

Issues 

The different requirements for each Federal program must be reconciled and issues of 
confidentiality data must be resolved. Furthermore, the state is seeking to balance state 
accountability needs with local information needs, so that services can be provided in an effective 
and efficient manner. 
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The working committee has suggested that the statistical tools used to analyze the measures 
be supported with state-level technical assistance and training. Furthermore, the committee 
members suggest that the integrated information system should be capable of producing data through 
an ad hoc query module and in a format that can be used as input to a stand-alone statistical analysis 
software. 

A decision has not yet been made about whether to develop performance standards against 
which the outcome measure should be judged. Four approaches for developing such standards have 
been suggested: 

1 . Use the first year that the program measures are implemented to establish.baseline data 
against which continuous program improvement will be judged rather than establishing 
standards a priori. 

2. Establish standards only after baseline data are collected. 

3. Establish “default” statewide standards and, subsequently, negotiate local standards with 
local staff according to local conditions and priorities. 

4. Establish a methodology, similar to the multiple regression model used for JTPA, to 
statistically adjust standards according to local conditions and demographic 
characteristics of participants served. 

North Carolina 

North Carolina’s efforts to develop a workforce development system originated with an 
Executive Order from the Governor. The initiative is being carried out by an interagency committee, 
the Commission on Workforce Preparedness. 

Methodology for Collecting Measures 

The project is using data from the state wage record systems and tapes from each program to 
assess labor market outcomes. Data are being analyzed for all participants, currently on a quarterly 
basis. It is expected that the timeframe will need to be changed to an annual basis because some 
programs, such as community colleges, cannot provide accurate data more frequently. 

Current Status 

The state is still in the process of finalizing the program measures for the system. The goal 
is to begin collecting data on the measures beginning July 1, 1997, the new program year, with the 
first data on the measures available in October or November 1998. The plan is to have the 
workforce performance management system measures in place for the 1999 program year. 

Human Resource Investment Council members have approved the measures and the data 
collection plans. Individual state agencies now are working toward mating the implementation 
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schedule. In addition, approval to request Social Security numbers of all high school students must 
be given and recommendations must be made for ongoing funding of a Common Follow-up System. 

The project is focusing first on labor market outcomes. Next, the return on investment 
measures will be considered. Outcome measures related to learning and skill gains are not expected 
to be finalized for at least another year. 

Issues 

At least three issues related to developing and implementing a workforce performance 
measurement system are being addressed by North Carolina. The biggest issue involves which 
participants should be included in the analyses. The feeling is that individuals who have not yet 
completed a program should be included in the analyses, if they have participated for a sufficient 
length of time to be influenced by the program. 

The accuracy of self-reported data about employment status for program participants is the 
second issue being examined. Although at present programs rely on such self-reported data, some 
thought is being given to using a wage test from the state unemployment insurance record data for 
program participants, since these data may more accurately reflect the employment status of 
program completers. 

The third issue being addressed is how to ensure that the data collected and analyzed are 
useful for local programs, as well as for state agencies. An effort is being made to determine what 
data local programs need and want, and to fit these information needs with what state agencies need. 
Specifically, there are questions about the data that should be collected related to courses of study 
and training programs. 

Texas 



Over the past four years there have been several workforce preparation initiatives in Texas 
that involve the development of participant outcome measures. In 1993, the Texas Council on 
Workforce and Economic Competitiveness, which was created by the Texas Workforce and 
Economic Competitiveness Act, organized interagency workgroups to develop core performance 
measures for the state’s workforce preparation system. Of the eight core measures developed by the 
Council, five have been approved on a preliminary basis, pending additional review and the 
Governor’s approval. 

Methodology for Collecting Measures 

Data will be collected from individual agencies that report common data elements, including 
the state’s unemployment insurance wage records. A data collection and reporting system, the Texas 
Automated Student and Adult Learner Follow-Up System, is being developed by the Texas State 
Occupational Information Coordinating Committee. It will combine data from state agencies and 
is being established from the follow-up of employers. 
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Current Status 

A two-phase plan for implementing a comprehensive workforce evaluation system has been 
approved by the Texas Council on Workforce and Economic Competitiveness, pending the 
Governor’s approval. During the first phase (through June 1997), additional research on the two 
performance measures is being conducted. These are: entered employment rate and the average 
earnings gain, based upon previous earnings. If approved, they will become the state’s key 
workforce outcome measures and will be included in the appropriations bills for all agencies. Two 
core performance measures would be eliminated: (1) the customer satisfaction measure at the 
state/systems level (although local agencies are encouraged to use them for local performance 
management systems); and (2) earnings gain based upon earnings at program entry. The basic data 
reporting elements for the measures also are being identified during the first phase. During the 
second phase, the remaining core measures will be refined, basic data reporting elements to support 
additional measures will be identified, and mechanisms to better communicate with local agencies 
will be developed. 

Issues 

The Texas Council on Workforce and Economic Competitiveness is continuing to work with 
its partner agencies in resolving concerns about access and equity, and in determining the 
relationship between program advancement and related measures. Research is being conducted on 
developing employer-oriented, skill attainment and cost-effectiveness (e.g., return on investment) 
measures. The Texas Council on Workforce and Economic Competitiveness also is working with 
state and local agencies to identify the basic data reporting elements required to support the core 
performance measures. 

Washington 

In developing a Washington state system, nine focus groups were conducted with staff from 
local program providers and program participants. The performance management system is intended 
to work at three levels: for the workforce development system as a whole, for population clusters 
served by more than one program, and for individual, statewide workforce development programs. 
Measures have been developed for four population groups: youth in-school, youth out-of-school, 
current and unemployed adult workers, and adults with barriers to employment. 

In the spirit of promoting continuous program improvement, it is expected that the system 
will change over time. Additionally, it is expected that, after baseline data have been collected and 
the performance management system has been in place for a number of years, each program will 
establish performance standards for the measures that are relevant to their program. 

State agencies will work in partnership with local program providers in developing the 
workforce performance management system. State agencies will provide local programs with 
information about best practices and offer support in the development of skill standards, information 
systems, and common data standards. Furthermore, state agencies will identify funds to support 
continuous program improvement and to implement program improvement plans. 
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Methodology for Collecting Measures 

As part of the implementation plan, it is expected that state agencies will collaborate on 
operationalizing the measures for each population cluster and in developing methodologies for 
collecting data on the measures, including the length of follow-up periods. 

Four levels of evaluation are planned: 

1 . System evaluations of the workforce training system by the interagency Washington 
Training and Education Coordinating Board (WTECB) that will assess the extent to 
which system goals have been met and to inform efforts to ensure continuous program 
improvement. 

2. Individual evaluations of the four clusters (i.e., youth in-school, youth out-of-school, 
current and unemployed adult workers, and adults with barriers to employment) by the 
WTECB. 

3. Individual agency evaluations by statewide agencies to measure the extent program goals 
have been realized and to analyze the relationship between program characteristics and 
program results. 

4. Local program evaluations to analyze the relationship between local program 
characteristics and local program results. 

Local program providers, as well as program participants, will collaborate on designing 
consumer reports. These reports will present information about program results, as well as other 
input and contextual information, such as the demographic characteristics of program participants 
and the availability of program resources. Such information will be made available to current and 
potential program participants. 

Current Status 

An implementation plan for the performance management system was scheduled to be 
developed during 1996 but has been delayed until Spring 1997. This plan will include indicators 
appropriate for the state program level, as well as consistent indicators for measuring the same 
outcomes, such as employment and earnings. 

Issues 

There is concern that the measures will not reflect what individual programs accomplish, as 
an individual measure cannot convey an entirely accurate picture of a program’s success. For 
example, a low unemployment rate can be the result of a population exodus from a locality, where 
the number of people employed is either constant or actually decreasing. Following this example, to 
obtain an accurate view of unemployment, data about other variables, such as total employment and 
population, must be collected. 
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The state makes clear that the fact that measures about which data are to be collected has 
been established is not intended to assume a direct cause and effect relationship between the 
performance of the workforce development system and the performance of the state on the measures. 

Performance Measurement, Reporting and Improvement System (PMRIS) 

Initiative of the National Institute for Literacy 

NIFL’s PMRIS Project began during the summer of 1993. The goals of the two year project 
were to plan, test and implement an interagency reporting and program improvement system that 
would promote continuous program improvement in adult literacy and basic skills programs. States 
were to develop interagency systems that would be “. . .based on a common understanding of how the 
major literacy and basic skills training programs operate in the state, and what they contribute to the 
achievement of broad state policy goals, such as strengthening families, reducing welfare 
dependence, and increasing workforce productivity.” The goal was that, under PMRIS, participating 
states would agree upon “...short-term and long-term goals for specific populations, indicators of 
program quality, and potential outcome and quality measures. . .( Swadley & Ziolkowski, 1996).” 

The system was to be developed through interagency policy teams representing education, job 
training and other social service programmatic areas. 

Specific objectives of the PMRIS project included: 

• State establishment of ongoing interagency policy and planning mechanisms that 
coordinate with state employment and training agencies, human resource investment 
councils, and adult and vocational education councils. 

• Documentation of the process for establishing interagency mechanisms. 

• Design of state performance measurement, reporting and improvement systems that are 
based upon a common understanding of how the major literacy and job training 
programs operate, agreement on short-term and long-term goals for participants in the 
different programs, indicators of program quality, and potential outcome and quality 
measures. 

• Development of long-term strategic plans for implementing an integrated performance 
measurement system, including detailed work plan schedules of activities, definitions of 
team members’ roles and responsibilities, and progress benchmarks. 

• The piloting and implementation of a performance measurement system. 

A cross-site evaluation of the PMRIS project (Swadley & Ziolkowski, 1996) reported the 
following lessons learned from the project related to building an interagency policy and planning 
mechanism: 

• Interagency collaboration is essential to project success. Support from the very top 
levels of government is necessary to surmount the hurdles that are inevitable. 
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• Broad stakeholder involvement from the beginning will better insure the project’s 
completion. 

• Interagency task groups working on the PMRIS must have a specific focus, authority, 
and uninterrupted time together to accomplish their mission. 

• States should develop a strategic plan to guide the development of performance 
management systems, which should be reviewed on a periodic basis. 

• Local participation from the very beginning of the developmental process is necessary to 
ensure that state and local priorities are linked together. 

• Efforts to collect data needed for performance management systems must be developed 
and the problem of state agencies not willing to share data must be addressed. 

• Data requirements should be directly linked to outcomes. 

• Adequate time and resources must be allotted for technical assistance. 

• It is realistic to expect that the development and implementation of a performance 
management system will take between three and four years. 

The five PMRIS states were: Alabama, Hawaii, Kentucky, New York and Tennessee, 
although funding for Alabama was only provided during the first year of the project. Tennessee and 
Kentucky are continuing with the PMRIS initiative even though NIFL funding has been completed. 
Hawaii developed measures through the PMRIS project, but is not continuing with the project. New 
York never reached the point of developing measures and the project has not continued since NIFL 
funding expired. 

Following are brief descriptions of the PMRIS project in the three states that developed 
measures during the two-year project: Hawaii, Kentucky and Tennessee. 

Hawaii 

Hawaii viewed the PMRIS “....as a system of shared accountability for the quality of 
policies, programs and practices which impact on the literacy service delivery (Swadley & 
Ziolkowski, 1996, p. 38).” The state’s participation in the PMRIS project began against a backdrop 
of strong public support for literacy activities. Most state agencies involved with literacy activities 
were members of the Governor’s Council for Literacy. The PMRIS grant was operated through the 
Hawaii Literacy Policy in Action Academy (HLPAA), representing a variety of public and private 
agencies and created just for this purpose at the suggestion of the Governor’s Council for Literacy 
and Lifelong Learning. Each organization participating in the HLPAA signed a contract specifying 
their responsibilities related to the PMRIS project. 
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Existing performance measures and data elements used by HLPAA agencies were reviewed, 
and subcommittees worked on developing policy outcome and performance measures. A system for 
collecting data has not been completed and staff involved in the Hawaii project report that creating a 
vision and outcomes was easy when compared with the task of developing a results-oriented 
information system. 

Agencies Involved 

Hawaii’s project involved an impressive number of agencies: Board of Education, 
Department of Budget and Finance, Department of Business, Economic Development and Tourism, 
Department of Education, Community and Special Education, Special Programs Management, 
Department of Human Services, Department of Labor and Industrial Relations, Department of Public 
Safety, Business Roundtable, Hawaii Community Foundation, Hawaii Community Services Council, 
House of Representatives, Hawaii Literacy, Inc., Hawaii Pizza Hut, Inc., Hawaii State Public Library 
System, State Teachers Association, State Senate, Hui Malama Learning Center, Kamehameha 
Schools/Bishop Estate, Office of Community Services, Office of the Governor, Office of the 
Governor’s Office of Children and Youth, Ogilvy and Mather, Pacific Regional Educational 
Laboratory, Office of Vocational Education, University of Hawaii’s College of Education and 
Hawaii Community Colleges. 

Outcome Measures 

The project defined a wide range of measures. 

• Economic impact. Percentage of employment in industries where the average annual 
wage was more than $22,000 in 1992 and income of participant. Three measures of 
income were proposed, GSP per employed worker, average annual wage and median 
household income. 

• Credentials. Percentage of students who attain a high school degree or the equivalent 
and percentage of people 25 years of age or older who attain a Bachelor’s degree. 

• Learning Gains. Percentage of students scoring above Level 1 on the Hawaii Statewide 
Literacy Assessment (HSLA). 

• Family Impact. Percentage of entering kindergarten students having attended preschool, 
number of parents who volunteer in school classrooms, number of parents participating 
in parent community network center parent-child interaction activities and number of 
parents who read to their children daily. 

• Community Impact. Percentage of eligible individuals who vote and percentage of 
registered voters who vote. 
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Methodology for Collecting Measures 

The PMRIS project ended before Hawaii reached the stage of developing a methodology for 
collecting measures. 

Current Status 



There have not been any ongoing activities related to the project since the two-year NIFL 
funding cycle was completed. The Hawaii Strategic Plan, however, indicates that demonstration 
sites were to be selected during 1996, but this schedule has not been followed and demonstration 
sites have yet to be selected. , 

Issues 

Hawaii struggled to address the following issues in developing an outcome-based 
performance management system. 

• Separate and distinct data systems in each state agency, which staff felt could not be 
changed. 

• “Right to privacy” laws that are more stringent than those of most states. 

• A lack of enthusiasm among the various state departments for sharing information. 

Kentucky 

Kentucky began the PMRIS project with adult education and literacy occupying a 
prominent position on the state’s policy agenda. In 1990, state legislation created the Cabinet for 
Work Force Development, which was to house and integrate adult basic education and job 
training programs within a single agency. Consistent with the objectives for the new cabinet 
workforce development agency, PMRIS was viewed by the state as an opportunity to simplify its 
literacy programs by developing an evaluation and information system that collects and reports 
consistent data on programs and learners. There were at least four reasons why Kentucky was 
interested in PMRIS: (1) to expand the state’s ability to have an outcome-based accountability 
system to monitor program quality; (2) to allow data to be collected and reported through an 
automated system; (3) to allow local programs to collect, enter and report on all data needed for 
Federal reporting purposes; and (4) to integrate all data systems with one another so that all state 
agencies would receive the information they need. 



The PMRIS proposal was prepared by Kentucky’s Department of Adult Education and 
Literacy, which became the state agency responsible for the project. Establishing PMRIS as an 
interagency effort, however, was reportedly easily accomplished because of the strong 
coordination existing between many state departments and agencies. Four interagency teams 
were established to conduct PRMIS: (1) outcomes — to refine the vision statement and policy 
and program outcomes; (2) stakeholder — to identify barriers to providing literacy services; 
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(3) measurement — to determine performance measures for policy and program outcomes and 
benchmarks for each outcome; and (4) systems development — to create a PMRIS database and 
identify the existing systems that can be integrated with PMRIS. 

Agencies Involved 

Kentucky organized an Interagency Policy Team, led by the Department of Adult Education 
and Literacy, with representatives from the commissioner level and higher. The following agencies 
were involved with Kentucky’s PMRIS project: Kentucky Job Training Coordinating Council, 
Department of Education, Workforce Development Cabinet, Department of Vocational 
Rehabilitation, Department of Libraries and Archives, Department for Social Insurance, Department 
for Employment Services, Department for Information Services, Department of the Blind, 
Department of Technical Education, Department for Adult Education and Literacy, Governor’s 
Office of Policy and Management, Office of Training and Reemployment, and the Legislative 
Research Commission. 

Outcome Measures 

Kentucky developed measures in all of the areas of the Columbia Framework, except for 
customer satisfaction. 

• Economic impact. The three measures of economic impact developed are: percentage 
of participants employed; percentage of participants with an employment goal who 
achieve the basic skills foundation required by employers; and percentage of participants 
with a high school credential who gain, retain or advance in employment within 12 
months of leaving the program. 

• Credentials. Percentage of adults who will gain enhanced academic or technical skills as 
evidenced by measured progress or a credential, and percentage of participants passing 
the GED within 12 months. 

• Learning gains. Percentage of adults who attend for more than 50 hours of instruction 
and show the equivalent of a half-year gain on a state-approved standardized assessment, 
or attain individually stated educational objectives. Kentucky also included a measure of 
health education learning: percentage of participants who have acquired competencies 
for wellness and healthy lifestyles. 

• Family impacts. Two family measures were developed for participants who are parents 
of school-aged students: percentage who participate in school-related conferences, 
meetings or activities; and percentage of participants who demonstrate the skills 
necessary to ensure that their children succeed in school. A third family measure, to 
apply to all participants, is the percentage of participants who demonstrate family life 
skills competencies. 
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• Further education and training. Percentage of participants who transition into 
additional education, training or the military within 12 months of obtaining a high school 
credential. 

• Community impact. Kentucky developed two measures of community impact: 
percentage of participants who demonstrate knowledge of services and resources 
available in their community and the ability to access them, and percentage of adults who 
will be actively involved in community and civic activities. 

Methodology for Collecting Measures 

The management information system developed through PMRIS combines and integrates all 
common data elements from relevant systems such as JOBS, JTPA and the Department for 
Employment Services. It is designed as a unified information system with a standardized screen and 
consolidated information. Student folders containing the data are stored on the state’s mainframe 
computer and local service providers having access to these folders. 

Current Status 

The reporting and information component of PMRIS are currently being used on a voluntary 
basis by approximately half of the local adult education programs (110 programs). These programs 
are transferring data to the state either by disk or online. It is Kentucky’s intention eventually to 
require all adult education programs to use the system and to tie all funding to performance on these 
outcome measures. 

Issues 

In conducting the PMRIS project, Kentucky faced a number of major barriers: 

• Federal and state legislative provisions impeded efforts at coordination, joint planning 
and consensus building. 

• Focusing on learner outcomes rather than program processes requires members of the 
PMRIS team to change dramatically the way they approach program accountability 
issues. 

• The lack of electronic networking between state agencies hindered efforts to establish an 
electronic information system that would allow data to be shared. 

• The length of the PMRIS project was not sufficient to implement fully the project’s 
objectives. 



Tennessee 

Tennessee’s PMRIS project, known as the Tennessee Reporting and Improvement 
Management System (TRIMS), was an outgrowth of the^T^nnessee’ State Literacy Resource 
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Center’s (TLRC) effort to promote a seamless adult literacy system of high-quality services. TRIMS 
was intended to develop further the existing collaboration between state agencies offering adult 
education employment training services. A policy committee, established from members of the 
TLRC Advisory Committee, was created to design, implement and institutionalize TRIMS. Most of 
Tennessee’s effort with PMRIS concentrated on developing program outcomes. After PMRIS 
funding was completed, Tennessee received a grant from the U.S. Department of Commerce to 
develop further the performance accountability system through the Internet by promoting 
interagency data and linking of local service providers. 

The project was envisioned and implemented as a “bottom-up” effort, with task forces, 
consisting of representatives from local programs, established to develop models of how local 
agencies can work with one another in developing a common outcome framework with shared goals. 
Efforts to involve adult learners in the process of developing outcome measures were undertaken, 
through roundtable discussions and focus group interviews. Members of the TRIMS team, however, 
reported being disappointed in the reluctance of adult learners to participate in the process. Through 
a series of meetings, local task forces identified key target customer groups, specific program 
outcomes and the skills, knowledge and resources needed to achieve the outcomes. A subcommittee 
of the state policy team then worked with representatives from the local task forces to refine the 
outcomes and measures. 



Agencies Involved 

TRIMS involved representatives from state agencies and other organizations as coordinated 
by the University of Tennessee’s Center for Literacy Studies. Agencies and organizations 
represented on the interagency committee included the Office of Information Resource; 
Telecommunications Application Partnership, Institute for Public Service, University of Tennessee; 
Board of Regents; Department of Education, Division of Adult and Community Education; 
Department of Employment Security; Department of Human Services; Department of Labor; 
Literacy Coalition; State Library and Archives; Tennessee Tomorrow, Inc.; and four local partners: 
Haywood, Knox, Lawrence and Wilson counties. 

Outcome Measures 



Outcome measures in five areas related to the Columbia Framework have been developed: 

• Economic impact. Number of participants who earn annual wage increases, number of 
participants who do not receive unemployment benefits, number of participants who earn 
acceptable performance reviews, number of participants placed in jobs, participants’ 
hourly wages, benefits and length of employment. 

• Credentials. Number of participants receiving a GED, external degree or an adult high 
school diploma; and the number of participants receiving skills certificates, such as 
WorkKeys certification, Associate Degrees, and technical certification. 

• Learning gains. Improved basic skills as measured by pre- and post-tests, and number 

of participants who achieve their educational goals. 4 | 
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• Family impact. Number of participants having good life-coping skills. 

• Economic impact. Number of participants no longer receiving welfare; number of 
participants with income above the poverty level. 

• Community Impact. Number of participants who gain U.S. citizenship, number of 
participants who register to vote. 

Methodology for Collecting Measures 

When it became evident that the data needed to measure program outcomes were not 
available, a Technology Committee was established to consider a state tracking system rather than 
the existing system of county-level sharing of information. A prototype system was developed to 
demonstrate the feasibility and value of sharing information at the local level. Using the state E-mail 
system, the prototype created a single record for each participant with all agencies able to add to, and 
access, the system. With only 10 data fields and a text system rather than a database, each agency 
would have to enter client data and, therefore, it is not usable on a large-scale basis. The system then 
was replicated using the state mainframe, with 50 datafields and with a reporting capacity. A 
prototype comprehensive outcome-based data base has been made available to local programs 
through the Internet, giving them access to the data at all times. 

Current Status 

As a result of TRIMS, a management information system that meets the data collection 
requirements and needs of different state and local agencies is being pilot tested at four local sites. 

Issues 

The major issues in developing a performance measurement system identified by people 
involved in the project are to: 

• Remember that planning to collect data is not the ultimate object. It is essential to have a 
clear understanding, at the beginning of the project, of how performance measures are 
going to work and benefit participants. Performance measurement is only a part of a 
large system of performance accountability and continuous improvement. 

• Consider the issue of technology at a project’s outset. 

• Allocate sufficient time and resources to design and implement a performance 
management system. 



State Initiatives 



To identify other projects focused upon developing outcome measures for adult education, 
we asked the state adult education directors for information on their current reporting activities and 
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to describe any other ongoing initiatives in their states in which adult education was involved. We 
received responses from 35 states. 

While all states reported collecting at least the information required for the DAEL Annual 
Statistical Report, many states collected additional measures for in-state use, summarized in Table 5. 
Kentucky collects the broadest range of additional measures, including family and community 
impact measures, as well as attainment of employment-related thinking skills. These measures were 

TABLE 5 

Additional Learner Outcome Measures Collected by States 
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Outcome Domain S 
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. j Measure # [''f' 


State vj.V' 


Family Impact 


Demonstrate skills needed for family life 


Kentucky 




Demonstrate skills needed to promote children's 
success 


Kentucky 




Parents volunteer in school 


New York 




Parents involved in school improvement student 
behavioral changes 


New York & 
Minnesota 


Community Impact 


Demonstrate skills needed to use community 
resources 

Participate in civic activities 

Demonstrate skills for wellness, healthy lifestyles 


Kentucky 

Kentucky 

Kentucky & 
Massachusetts 




Obtain a library card 


Kentucky & West 
Virginia 


Further Education and 
Training 


Number of students meeting with staff to discuss 
education and career plans 


Illinois 




Continue to college or university 


Alaska 


Economic Impact 


Demonstrate basic thinking skills required by 
employers to advance in or keep a job 


Kentucky, New York 
& Oregon 




Participate in work or community learning 
experience 


Oregon 


Customer Satisfaction 


Student's perception of usefulness of basic skills 
instruction 


Washington 


Learning Gains 


Pre/post scores or gain scores 


Illinois, Indiana, 
Maryland, Mississippi, 
Missouri, 

New Hampshire, 

North Dakota & 
Pennsylvania 




Demonstrate skills necessary to utilize banking 
services 


Kentucky 




Competencies attained 


Colorado & 
Washington 
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developed through the state’s PMRIS project. New York also is collecting family impact measures 
and measures of employment-related thinking skills. Illinois and Alaska collect additional measures 
of employment and training, and Washington collects measures of students’ perception of the 
usefulness of instruction, and is the only state that collects a customer satisfaction measure. Eight 
states reported collecting pre- and post-test scores from local programs and Colorado and 
Washington also obtain the competencies attained by learners. 

In addition to collecting other types of measures, several states reported new initiatives 
underway, in which adult education was involved. In the area of workforce development systems, 
adult education staff in five states — Maryland, Idaho, North Carolina, South Dakota and Alaska — 
are developing common outcome definitions, data collection plans and student follow-up methods 
with their respective governmental agencies or commissions on Workforce Development or Human 
Services. 

Pennsylvania, Illinois and Kentucky have begun programs to improve and validate their 
current data collection system and to improve the quality of student outcome data. These projects 
seek to standardize definitions, develop clearly defined indicators for student learning and 
implement better collection processes. Through Project Equal, Pennsylvania is working toward 
standardizing the quality of assessment at the local level and improving the information reported to 
the state. Illinois is now requiring all local programs to report pre- and post-test scores and may 
eventually develop performance standards for student learning gains that will be tied to future 
funding. Kentucky is improving its outcome data collection by adapting the methods and measures 
developed through its PMRIS project, described above. 

Below we describe four major state initiatives that can best inform the development of a 
national outcome reporting system for adult education. In California, adult education is involved in 
a state-initiated, multi-agency effort to improve reporting of all programs involved in workforce 
development. Florida is operating an interagency data collection system that involves adult 
education, although to a limited extent. Adult education offices in two states - Oregon and Texas - 
are conducting major projects to develop or improve their outcome systems. 

California 

A recently enacted state law (SB 645) requires that the State Job Training Coordinating 
Council (SJTCC) design and implement an outcome-based system to measure the performance of 
state and Federally funded education and training programs that comprise the state workforce 
preparation system. The law also requires that the SJTCC use the performance data from the system 
to prepare a series of “report cards” on all adult education and training programs, local and state 
workforce development programs and the state workforce training system as a whole. The SJTCC 
established the Performance-Based Accountability (PBA) Committee to develop the system by 
defining measures to be collected, determining a methodology for collecting them and developing 
implementation plans. The Committee also is identifying the audience for and establishing the 
content of the proposed report cards. The implementation will take five years and proceed in two 
phases, with the first phase focusing on employment measures. During the second phase, the PBA 
Committee will refine the Phase 1 measures and develop measures for knowledge and skill 
attainment. Full implementation is planned for 200 1 . 
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Agencies Involved 

The agencies that will be part of the system and that have representatives on the PBA 
Committee include adult education, vocational education, JTPA, JOBS, community college 
programs, Job Service, Food Stamp Employment and Training Programs and the state Employment 
Training Panel. 

Outcome Measures 

The PBA Committee identified the goals for the workforce development system and then 
proposed seven measures addressing the goals. 

• Economic impact. The proposed Phase 1 economic impact measures are: 

1 . Job placement rate; 

2. Wages and earnings, measured by wage rates, total wages, and earnings before, 
during and after participation (i.e., immediately, one year, three years); 

3. Length of employment; and 

4. Rate at which welfare recipients become employed. 

• Learning gains. Percent of participants that achieve workplace skills. 

• Further education and training. Rate of entry into public, post-secondary education. 

• Customer satisfaction. Employer satisfaction with the training of participants. 
Methodology for Collecting Measures 

The PBA Committee plans to use each program’s existing data base and not make substantial 
changes to current reporting systems at this time. The development of the report cards is primarily a 
data analysis activity. Using Social Security numbers, information on participants from the program 
data bases will be matched against the Unemployment Insurance, Social Security Administration and 
university and college systems, to obtain the economic impact measures — employment, wages and 
earnings — and entry into higher education. Beginning in 1998, employment retention also will also 
be measured in this way. Learning gains will not be measured until Phase 2, to give the committee 
time to determine the skills to assess and to establish common assessment methods. A survey, also 
to be implemented in Phase 2, will measure employer satisfaction. All measures may be revised in 
Phase 2, based upon the experience in Phase 1. 
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Current Status 

For the first report card, due in December 1997, the PBA Committee asked for a 
representative sample of each of the workforce development programs to provide data. Program 
representatives proposed the programs they would include to the Committee, which then decided 
which to include. In each of the next four years, more programs will be included until there is full 
implementation, scheduled for 2001. The Committee is currently negotiating for a contractor that 
will develop a data system and conduct the analyses. 

Issues 

Three major issues currently confront the PBA Committee as it develops the system. The 
Committee must first develop software to extract the data elements from the different systems and 
conduct the analysis. Three options are being considered - adapting a similar system used in 
Florida, adapting a system used by California’s Community College system, or developing a new 
system. 



A second problem is that education programs in California are not required to obtain Social 
Security numbers from participants, yet the system will require these numbers to link records across 
data systems. There is some controversy in the state as to whether education programs can require 
students to give their Social Security numbers, but all agencies will have to report them, if the 
system is to be implemented fully. Finally, some of the participating agencies have expressed a 
broader concern about the legal ability to share any confidential data with other agencies. This legal 
issue also will have to be resolved prior to full system implementation. 

Florida 

The state of Florida has established the Florida Education and Training Program (FETPIP), 
an interagency data collection system that obtains follow-up data on former students and provides 
this data to the programs and organizations in which the students received services. The purpose of 
this data collection system is to provide an analysis of program effectiveness. All participating 
agencies provide FETPIP with individual student or participant files from their respective 
management information systems. The files include individual identifiers such as names, Social 
Security numbers and demographic, socio-economic and programmatic data. 

Agencies Involved 

The Florida agencies currently involved in this system include the Department of Education, 
the Department of Labor and Employment Security, the Department of Health and Rehabilitative 
Services and the Department of Corrections. The specific program entities in these agencies from 
which data are collected include all public high school system graduates and dropouts, all 
community college associate degree and vocational students, all secondary and postsecondary 
students, state university graduates, adult education and GED students, selected private vocational 
schools, colleges and universities, JTPA programs, Project Independence participants (welfare 
clients), unemployment insurance claimants, and correctional system releases. 
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Data are obtained on the following three areas included in the Columbia Framework: 

• Economic impact Government job status, including job classification and pay grade, 
wages, length of employment of clients, and welfare participation or reduction after 
program participation. 

• Further education and training. Enrollment in other education programs and enlistment 
in the military. 

• Customer satisfaction. Employer satisfaction with training of program participants. 

Methodology for Collecting Measures 

Information on individual students or participants is provided to FETPIP from each 
program’s MIS. Using Social Security numbers, the program matches participant files 
electronically by linking them to the administrative records or other state and Federal agencies. 
Data are linked annually, although employment records are updated quarterly. The customer 
satisfaction measures are collected through an annual survey of a sample of employers who have 
program participants. 

Current status FETPIP has been operational since 1993 and results have been used by the 
involved agencies for planning and evaluations. Currently, the system has over 1.3 million 
participant records. 



At least four issues are being addressed by FETPIP. The most pressing issue facing Florida 
is defining what it means to complete a program. A key question is whether analyses should be 
limited to programs completed, since it is assumed that at least a portion of “non-completers” should 
have received some benefit from their participation. A second issue involves finalizing what data 
from individual programs should be reported, in addition to basic demographic information. 

The accuracy of self-reported data describing employment status is a third issue being 
investigated by FETPIP. There is some thought that the state should use unemployment insurance 
records rather than self-reported data by program participants. The fourth issue involves how to 
ensure that data being collected are useful to local agencies and how such data fit with the state-level 
data needs. 



Education, job training and social service agencies in Oregon have worked together since 
1988 to established coordinated service delivery, and to develop data sharing arrangements and 
common outcome measures. Begun in response to the 1988 welfare reform legislation, state 
representatives of the agencies formed the State Partner Leadership Group, as a mechanism for 
coordinated planning. Over the last two years, the Group has developed performance measures for 
workforce and education programs and plans to have full implementation of these measures by 1999. 



Issues 



Oregon 
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Agencies Involved 



The State Partner Leadership Group is composed of policy analysts and practitioners from: 
the Office of Community College Services, the main provider of adult education in Oregon; the 
Employment Department; Adult and Family Services; Department of Corrections; Job Training and 
Partnership Act; Economic Development Agency; Office of Professional and Technical Education; 
and the Oregon Workforce Quality Council. 

Outcome Measures 

The Group is developing 13 measures for workforce performance programs, which are still 
being finalized and are subject to continual refinement. 

• Economic impact. These measures are still being specifically defined, but will include 
number of participants who entered employment, job retention after 90 days and 
earnings. The return on investment also will be computed, by estimating the amount 
spent on education and training and return to the state in terms of productivity and taxes. 
Measures of economic self-sufficiency also are planned and will include reduction in 
welfare caseload and welfare recidivism. 

• Learning gains. Three measures of skill gains have been proposed, a general 
educational measure and two measures of workforce preparation. 

1 . Number and percent of learners who increase proficiency levels in reading, 
writing, math, speaking and listening. The functional levels correspond to the 
Federal definitions and are tied to performance descriptions: reading and math 
CASAS skills, SPL levels for oral proficiency and a holistic writing scale that 
corresponds to the GED 1-6 scale. 

2. Number and percent of learners who become workplace ready, defined as 
demonstrating a competency in workplace readiness skills. The specific skills 
are being refined and include SCANS skills. 

3. Number and percent of participants who complete an integrated 
work/community-based learning experience, such as a mentoring, job shadowing 
or youth apprenticeship. 

• Credentials. Number and percent of learners who complete an educational level or 
certificate, such as a high school diploma, GED, higher education diploma or certificate 
or industry specific certificate. 

• Customer satisfaction. Employer and client satisfaction with services. 
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Methodology for Collecting Measures 

Multi-agency data collection and sharing is planned for the measures, whereby each agency 
collects the information relevant to its program and data then are combined at the state level for 
computation of measures across agencies. The Oregon Shared Information System (SIS) was 
developed for this purpose, specifically to allow computation of the economic measures. 
Consequently, SIS is run by the Employment Department and overseen by the Oregon Workforce 
Quality Council. Adult education programs will eventually submit data to SIS and student records 
will be matched using Social Security numbers to develop the employment-related measures. 
Currently, only JTPA programs will collect the customer satisfaction measures, although other 
agencies may collect these measures in the future. 

Adult education programs are responsible for collecting the learning gains and credential 
measures. Programs administer pre- and post-tests using CASAS or appropriate assessments that 
measure functional literacy. Programs also report the number of students who attained credentials 
and other achievements each year. To automate data collection, all adult education programs will 
use the Tracking of Programs and Students (TOPS) system. Data will be included in the SIS and 
other programs’ accountability systems. 

Current Status 

The state is still refining many of the outcome measures based upon pilot tests by local 
programs. The SIS measures are being computed for JTPA, JOBS, and UI; vocational rehabilitation, 
adult education and other agencies will be added to SIS over the next three years. Adult education 
programs are currently collecting the learning gains and outcome measures and have just completed 
a pilot test of TOPS. Use of TOPS by all programs is planned by 1999. Methodologies are still 
being worked out for many measures, including customer satisfaction, return on investment 
measures, and SCANS skills. 

Issues 

Development of the integrated service and accountability system has taken a great deal of 
time and resources and requires continual change and refinement. It has required constant 
reaffirmation and support from all agencies to continue. A “win-win” orientation of all agencies 
involved has been key to its success. Support and involvement at the local level also must be built 
by demonstrating the value of the data to meet the needs of teachers, students and case managers. 

Texas 



The Texas Performance Measures Assessment System (PMAS) initiative will develop a 
statewide assessment system for determining literacy proficiencies and student progress in adult 
education. The Texas Education Agency is developing the PMAS specifically for students who plan 
to participate in, but lack the literacy skills to enter, the workforce development system. 
Consequently, both adult education and employment programs, such as One Stop Centers, will use 
the assessment. The planned assessment system will be based upon the framework used for the 
Texas Adult Literacy Survey, conducted as part of the National Adult Literacy Survey (NALS) and 
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will incorporate the prose, document and quantitative literacy scales. Unlike the NALS, however, 
writing, speaking and listening proficiency scales will be included. The PMAS will include an 
initial assessment or screening test, a baseline assessment upon entry in the adult education system 
and progress assessments during participation in adult education. 

Outcome Measures 

Student outcome measures are still being identified. The measures currently collected for 
DAEL reporting, including advancement to another education functioning level, will continue to 
be included. The method for assessing gains is described below. 

• Learning gains. The PMAS will create a single, uniformly implemented assessment 
method for determining the progress of adult education students. Like the NALS upon 
which it will be based, the assessment will have five proficiency levels for each area 
assessed. As the primary objective of the adult education is to move students from 
Levels 1 and 2 to Level 3, the project is focusing on defining proficiencies in these two 
levels, each of which have been subdivided into three additional levels (i.e., 1A,1B, 1C, 
2A, 2B and 2C). 

Methodology for Collecting Measures 

The PMAS is being developed for use in adult education, including adult education offered 
as a part of the workforce development system in the state. Career Development Centers (Texas’ 

One Stop Centers) may administer the initial intake assessment as a basis for referral. Clients 
scoring below Level 3 will be referred to adult education for further baseline assessment. Clients 
scoring Level 3 or above who do not have a high school credential will be referred to adult 
secondary education or high school equivalency testing. Clients scoring Level 3 or above who have 
a high school credential will be referred to job training and placement. 

Current Status 

The PMAS project is into its second year and a validation study of the initial intake 
component is underway. A sample of adult education programs, One Stop Centers, JOBS and JTPA 
programs volunteered to participate in the validation. A full field test of the instrument is planned 
for early 1997. Development of the baseline and program assessments also will begin in Spring 
1997. 



Issues 

One significant issue for Texas is that adult education’s role in workforce development is 
still unclear, especially in light of recent Federal welfare reform legislation. Other issues include the 
diversity of literacy needs, including the linguistic diversity, in the adult education and literacy 
population, the appropriate assessment procedures for undereducated adults, the fact that assessment 
drives instruction, the training needs for successful implementation, and the costs of developing the 
assessment system. 
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Summary of Measures, Methods and Issues 

Table 6 presents a summary of the outcome measures used in the reporting systems and 
projects reviewed in this chapter, along with their methodologies and the issues they have 
encountered in data collection or development of outcome measures. 

Two strong trends emerge from this review. First, there is a preponderance of measures on 
the economic impact of programs, both for participants and for society as a whole. All of the 
systems reviewed here include measures of whether participants obtain jobs, are removed from 
welfare, remain in their jobs, improve their job situations or develop skills that will help them 
succeed in the workplace. Many systems also include broader, systemic measures of economic 
impact by calculating program costs, welfare costs and participant increased earnings and taxes paid, 
to estimate net program impact in terms of public investment. 

A second trend is the move toward interagency collaboration and accountability. The two 
major demonstration projects, NGA and PMRIS, place adult education and training as components 
of a broader state system, and seek to develop accountability based upon each program’s role in that 
system. The breaking down of categorical program barriers for both service delivery and 
accountability, is the hallmark of this approach. 

The methodologies used to collect measures among the programs fall into three categories. 
Direct program reporting, the most common approach, relies on the service provider to keep track of 
participant outcomes and report them periodically to the state. This approach is sometimes 
supplemented with follow-up surveys of participants a short time after they leave the program to 
obtain information about outcomes. A data-sharing approach is planned for many programs with a 
workforce orientation, which involves pooling of data from several agencies. Analyses are 
conducted with the pooled data to evaluate program impact on employment, income and further 
education measures. 

While much thought and considerable time and resources have been invested in program 
accountability, most of this work is very new and there are very few accountability systems in 
operation. Only two national reporting systems for collecting outcome data on adult participants 
exist — JTPA and DAEL’s Annual Statistical Report — and among the new state systems, only 
Florida’s FETPIP and Illinois’ Common Performance Management system, are operating fully. The 
other projects are in various stages of development. 
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‘'Develop in g a National Outcome Re porting System for th eAdult Education Program 

CHAPTER 3 
Seven Key Issues 

Although outcome measures are correctly viewed as the central component of a performance 
management system, they represent the end product of a set of decisions on fundamental issues 
related to the program’s policy and direction. Policymakers, program participants and the general 
public will judge the program according to performance on the measures. In addition, the nature of 
the services offered by the program often are affected by the outcome measures. Consequently, 
considerable thought should be given to the assumptions and program goals that serve as the 
foundation for the measures before selecting them. 

This chapter presents seven key issues that adult educators must consider as they work 
toward developing an outcome-based reporting system for their program. To aid in promoting 
thought and discussion as the system develops, we have written the issues as a series of questions: 

1 . What policy and performance goals will the reporting system address? 

2. Who are the reporting system’s audiences or customers ' ? 

3. What outcome measures should be included in the reporting system? 

4. What methods should be employed to collect the measures? 

5. What will be the role of performance standards in the reporting system? 

6. How can unintended effects of performance measurement be minimized? 

7. What staff development and infrastructure are needed to operate the system? 

Within each question, we present the major topics to consider for the system’s development. 

As Figure 1 shows, these questions fall into three categories: policy issues, measurement 
issues and implementation issues, and have a logical order (e. g., policy issues need to be resolved 
prior to measurement issues). During their first meeting on the reporting system this March, adult 
education stakeholders are not likely to resolve these issues completely and, indeed, their resolution 
will probably evolve over time. However, some immediate decisions and thought about these issues 
will allow the development of the reporting system to proceed. 

As there has been considerable effort over the last several years in developing performance- 
based accountability systems, the discussion of these issues draws from the experience and lessons 
learned from these other projects. In particular the work of the NGA (Trott and Baj, 1996), NIFL 
(1995) and the U.S. Department of Labor (1994; 1996) inform the discussion of issues here. Chapter 
2 describes the projects sponsored by these agencies. 



68 

Chapter 3 : Seven Key Issues , J 4 ' * ■ : v . 52 



'Deye^pjhg a National Outcome Reporting . System jfor the 4&dt Eduction trogram 

FIGURE 1 

Major Issues in Development of an Outcome Reporting System 



Policy Issues 




Measurement Issues 


• Policy and Performance Goals 


► 


• Outcome Measures 


• Audiences and Customers 




• Methods 






• Unintended Effects 






• Performance Standards 



Implementation Issues 

• Infrastructure 

• Staff Development 



Issue 1 : What policy and performance goals will the reporting system address? 

An outcome-based accountability system is designed to gauge a program’s success toward 
meeting its policy goals. These goals define why the program exists. Legislation or policy 
directives typically outline a program’s purpose and goals and these documents are the first sources 
to consult when determining goals. What makes performance assessment and evaluation so difficult 
in adult education is that the Adult Education Act describes very broad and ill-defined goals for the 
program. 

The wide range of measures in DAEL’s Annual Statistical Report, and other efforts to define 
adult education’s purpose, reflect this broad program emphasis. For example, NIFL’s Equipped for 
the Future (EFF) project identifies three roles for adult learners — worker, citizen and family 
member — and, through a survey of adult learners, defined four purposes of literacy instruction for 
learners: to gain access to information, to give voice to ideas, to make decisions for independent 
action and to build a bridge for future learning (NIFL, 1995). Additionally, California recently 
identified the purpose of adult education as enhancing the learner’s role as worker, in the 
community, in the family and to promote lifelong learning. 

The measures in the Columbia Framework reflect an almost identical, multi-faceted view of 
the goals of adult education instruction. As Table 7 shows, the measures imply four purposes of 
adult education: 



• Promote educational improvement and lifelong learning (including English language 
acquisition); 

69 

• Enhance employability and economic self-sufficiency; 
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• Improve civic participation; and 

• Improve family life relating to education and literacy. 



TABLE 7 



Purposes of Adult Education 

Implied by the Columbia Framework and DAEL's Annual Statistical Report 



cl 

. Purpose 


„ •••>- -..5 -- -- - 

Measure Type 


Lifelong Learning and Educational 
Enhancement 


• Learning Gains 

• Credentials 

• Further Education and Training 


Employment Enhancement 


• Economic Impact 

• Credential 

• Further Education and Training 


Citizenship 


• Community Impact 


Family Life Enhancement 


• Family Impact Measures 



These four purposes correspond to the roles for adult education instruction identified by NIFL and 
California and are consistent with the broad goals of the Adult Education Act, as well as new 
proposed legislation, such as the Adult Basic Education for the 21 s ' Century Act. The decision that 
must be made is whether the outcome reporting system will measure program performance in all of 
these areas or some subset of them. 

Recently, the U.S. Congress, through the Careers bill and other legislative initiatives, sought 
to redefine adult education as a component of a workforce development system. Several states also 
are moving in this direction. This view of the program would make adult education employment- 
focused, requiring an emphasis on economic impact and employment measures. Adult educators, 
however, have consistently fought to keep adult education an education program, which would make 
learning gain measures most appropriate for assessing the program’s performance. Adult educators 
appear to have been successful in defining the goal of the program as educational, and the current 
Congress is not moving toward workforce consolidation. However, the close link between adult 
education and employment programs makes it likely that any outcome-based reporting system will 
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need to include both types of measures. As this example makes clear, decisions on outcome 
measures represent nothing less than a statement about the purpose of adult education. 

Short- and Long-term Performance Goals 

Performance-based reporting systems are oriented toward measuring whether the program 
achieves its long-term goals. It is rarely possible, however, to measure long-term goals without 
investing substantial resources in evaluation. Consequently, although some outcome-based reporting 
systems include measures assessing long-term goals, most measures are focused on whether short- 
term goals are met. For example, a long-term goal of adult literacy programs may be to allow 
lifelong learning of participants. While measuring whether a participant is a lifelong learner poses 
substantial difficulties, learning gains or advancement in post-secondary education, short-term 
outcomes, can be measured more easily. 

Once the long-term goals of the program are agreed upon for performance assessment, short- 
term goals for each measure need to be delineated. These goals should be predictive of the long- 
term goals, which requires assumptions about how the two types of goals relate to each other. The 
relationship among the goals may be self-evident, or based upon research or practitioner knowledge. 

Issues 2: Who are the reporting system’s audiences or customers ? 

Outcome measures indicate a program’s goals and what it accomplishes. Several different 
audiences may want this information, but their specific interests in the program vary. Furthermore, 
different audiences will use the information from the system in different ways. Along with 
determining program goals and purposes, adult educators need to consider who the consumers of the 
reporting system’s information will be and how these consumers will use the information. 

From the two main purposes of the National Outcome Reporting System outlined in the 
Columbia Framework (see Table 1, Chapter 1) — program accountability and program improvement 
— at least four audiences can be identified: 

• legislators and policymakers at the Federal, state and local levels, 

• employers, 

• potential and current adult education students, and 

• program managers and practitioners. 

Each of these audiences has its own reasons for wanting to know about the adult education 
program’s performance. Legislators, for example, want to know whether the program is worthy of 
the public investment and continued funding. Recently, this has translated into a focus upon whether 
adult education can enhance the employability of its students or the students’ role as family members 
(e.g., through welfare reform). Employers typically want information on the skills program 
participants have when they leave the program, particularly workplace skills. 



ERJC ^kjLid-2 •• '** * ± k - ^ ^ Chapter3: Seven K ey Is sue s i 




Developin g a National Owfcomg Reporting Sy stem_ fdr the AdpU Education Progrprn , 

Adult education students have very different information needs and uses for performance 
information. They may want to know whether a particular program can help them achieve their 
personal education and employment goals, such as whether the program can help them develop 
functional or survival skills, pass the GED or obtain a job. Program managers may use performance 
data to help them identify successful programs and what works to help students. 

Issue 3: What outcome measures should be included in the reporting system? 

The task of selecting measures is simplified after making decisions on policy, performance 
goals and audience. Although the state adult education directors, through the Columbia Framework, 
have already made initial selections about the type of measures the reporting system should have, 
they will now revisit these decisions more formally. Six principles identified by Trott and Baj 
(1996) will assist meeting stakeholders as they consider measures. 

• Measures should have a clear relationship to system goals. 

• Measures should inform evaluative, planning and policy decisions. 

• Measures should be meaningful and easy to understand. 

• Care should be taken to guard against unintended consequences of the measures. 

• Performance should have a clear and direct effect on the measures. 

• Performance should be a primary influence on measures. 

We have already discussed the importance of the first two points, ensuring that measures 
reflect goals and policy decisions, and that audiences find the measures meaningful and easy to 
understand. The last two points on the relationship of performance to measures emphasize the need 
for a clear, direct link between the measures and instruction and other program services and 
activities. For example, if the purpose of adult education is to improve literacy skills, learning gain 
measures, test scores or skill attainment are appropriate measures. In a family literacy program, 
measures of parent-child literacy behaviors, such as parents reading to their children, also are clear 
and direct measures of the impact of instruction. 

The need to link instruction and accountability is one of the goals behind current national 
and state efforts to develop content standards for adult education. For example, NIFL’s EFF project 
is developing consensus within the field on what adults need to know in relation to the roles of 
parent, citizens and workers. EFF plans to support programs efforts to translate these skills into 
content standards, and to tie them to accountability measures (Stein, 1996). California, Florida, 
Oregon, Rhode Island, Wisconsin and several other states, also are conducting major efforts to 
develop curriculum standards (Kutner, Webb and Matheson, 1996). 

As a program should only be accountable for what it can accomplish, program instruction 
and activities should be a primary influence on measures. The more outside influences can affect the 
measure, the less desirable it is for assessing whether the program is achieving its goals. For 
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example, several proposed and current performance systems of adult education include measures of 
civic functioning, such as registering to vote and increased involvement in community life. These 
activities may indeed be a desirable goal for adult education, but unless the instruction and program 
activities directly target these goals, the measures may not be an appropriate means forjudging the 
impact of the program. Many aspects of an adult education student’s life beyond the control of the 
program may affect whether a person chooses to vote or to be involved in the community. 

Economic impact measures are a second example. Many workforce development projects 
propose to measure their project’s impact by assessing reduction in welfare recipients. This measure 
is subject to economic conditions, participants’ family situations, personal problems and many other 
outside influences that an employment or education program cannot influence. Consequently, these 
types of measures should be carefully considered prior to adoption to ensure that programs will not 
be held accountable for circumstances over which they have minimal or no control. The JTPA 
program experienced such difficulties with some of its original performance standards (U.S. 
Department of Labor, 1996). 

Issue 4: What methods should be employed to collect the measures? 

The discussion of methods involves deciding how data on the measures will be collected. 
Both measures and methods are often discussed together, as types of measures frequently have 
implications for a particular methodology. Some accountability systems require multiple 
methodologies, due to the inherent difficulties and complexities in collecting data. Our review of 
existing and planned accountability systems in Chapter 2 identified three types of methodologies 
currently used that can be considered for the adult education outcome reporting system. 

• Direct program reporting. In this most frequently used method, local programs directly 
measure and report all data on their participants using protocols or forms developed by 
the state or Federal agency overseeing the program. For example, JTPA programs 
collect and report data on participants, including whether they obtained employment. 
Adult education programs measure student educational functioning level and report any 
advancement in functioning over the program year. 

• Participant follow-up. Often used in conjunction with direct reporting, this method 
involves locating all or a sample of participants after a period of time to collect longer- 
term measures by mail, telephone or through personal interviews. Either the program or 
a third-party contractor conducts the follow-up. JTPA programs must collect follow-up 
data on at least a sample of participants 13 weeks after they leave the program. 

• Data sharing. In this method, each program collects and reports measures related to its 
goals, and data are pooled into a shared data system. The state then conducts analyses to 
assess progress toward performance goals. This method allows an individual to be 
followed through multiple programs and is used to measure participant employment and 
income gains for computing net program impact. Florida and Illinois use this 
methodology and many states, notably California, Kentucky and Oregon, plan to 
implement this procedure in the future. 

73 




w: 



Cha pter 3: Seven Key Issues ; 



developing a Ow /co/w e Reporting Syffim for the gtfacatfo w Progra m 

Several other methodologies are possible, such as a participant survey, where an outside 
evaluator could select all or a sample of participants to whom a survey or a test is administered that 
includes outcome measures. Many workforce programs are considering survey methodology to 
collect customer satisfaction measures, but this methodology is not typically used for reporting 
purposes, due to its high cost. 

Significant difficulties confront the implementation of any methodology and often a pilot 
test is tried to gauge the feasibility of a methodology, or to fine tune it prior to wide-scale 
implementation. One option when developing a new reporting system for adult education is to try 
one or more methodologies at the local program level before implementing them on a state or 
national level. 

Technical Issues 

Selecting an appropriate methodology usually requires consideration of statistical and other 
technical issues, as well as resources available for data collection and analysis. While a full 
discussion of these issues is beyond the scope of this paper, we present some issues that immediately 
confront the selection of methodologies for a national outcome reporting system for adult education. 

• Reliability. Reliability refers to whether measurement will be the same when conducted 
over time and by different people. For example, a perfectly reliable test, administered 
properly, will give the same result no matter who administers it and no matter where it is 
given. Reliability is particularly important for direct reporting methods since many 
different individuals - local program staff- must collect the measures over time. 

• Validity. Validity refers to whether an indicator actually measures the underlying event 
or property it purports to measure. Simpler measures, with a direct connection to the 
concepts underlying them, tend to have high validity. For example, whether a participant 
obtains a job is a valid measure of employment; improved reading ability is a valid 
measure of literacy gain. Weak methodologies, such as flawed sampling or insufficient 
follow-up, pose the biggest threat to validity of measures. 

• Sampling. A reporting system usually includes performance measures from all program 
participants. A scientifically drawn sample, however, can provide equally valid data 
with less cost. For this reason, evaluators often employ sampling methods, especially for 
measures that require participant follow-up. While highly valid sampling plans can 
usually be devised, successful implementation of a sampling methodology often is 
beyond the capacity of most program providers. Sampling methods require extensive 
knowledge of the characteristics of participants, rigorous procedures for drawing the 
sample and extensive follow-up efforts. Numerous potential selection biases can plague 
a sample, invalidating the data. 

• Data confidentiality and control issues. Methodologies that rely on data sharing, where 
multiple agencies collect and share information on the same participant, have to confront 
confidentiality issues. Some agencies have laws against sharing sensitive information 
(e.g., income, welfare status). In addition, this methodology requires a common 
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identifier across agencies, usually participants' social security numbers, which also may 
be difficult to collect and share. Data sharing arrangements also raise the issue of control 
of data — what is reported and by whom. States with this methodology have written 
interagency data sharing agreements that specify data access and control. 

• Resources. Collecting data is expensive and, often, reporting requirements are 
established without allocation of sufficient resources for program providers to meet 
them. In a recent pilot test of an adult education reporting system, local program staff 
cited lack of resources as a major reason they could not adequately report data (Condelli, 
1994). As greater resources normally produce better data validity and reliability, cost of 
data collection must be considered in designing a national outcome reporting system. 

Table 8 summarizes the three methodologies presented here along with examples of how 
the methodological issues affect their implementation. 

TABLE 8 

Three Data Collection Methods and Major Issues Affecting Data Quality 



Data Collection Method 


Major Issues Affecting Data Quality 

.. * • a *'*' ^"* **•' '' ' V' * 


Direct Program Reporting 


Reliability — Local staff may not measure 
consistently over time or over programs. 

Validity — Haphazard follow-up threatens validity. 

Resources — Requires local staff to devote 
resources to data collection and follow-up. 


Participant Follow-up 


Sampling — Requires good sampling and follow- 
up for valid data. 

Resources — Costly to conduct. 


Data Sharing 


Reliability and Validity — Depends upon ability of 
all programs to collect data. 

Resources — Requires shared data systems with 
common measures and student identifiers. 

Confidentiality and Control— One agency must 
control all data; confidentiality safeguards needed. 



Issue 5: What will be the role of performance standards in the reporting system? 



Measurement of effects inevitably invites evaluation and comparison. That is, it is not 
usually enough to describe a program's performance with outcome measures because audiences want 
to evaluate these measures against some standard of performance. The question becomes: Is the 




-’7' ■ . • • . • .•■•■v ; 'l ~ -y? : Cha pt er 3;^ Se ven Key Issues; 



' flgyg/o /w &a National Outcome Re porting S ystem , /frr rfre AduUEducati on Program 



program performing well enough? Setting performance standards is a way of defining “well 
enough.” With quantitative outcome measures, a performance standard defines acceptable 
performance in terms of a specific numeric criterion. 

Adult education stakeholders are still clarifying the policy goals, measures and methods for 
the outcome reporting system and, therefore, may not want to set explicit performance standards at 
this time. However, the issue cannot be ignored because, at some time, key audiences will demand a 
performance standard, or impose their own on the program (i.e., they will judge the program 
according to their own criteria). In addition, the performance strategy adopted can guide the types of 
measures and methods to select. 

The U.S. Department of Labor (1994) and Trott and Baj (1996) describe four types of 
performance assessment strategies currently used by the programs they reviewed. 

• Return on investment In this strategy the overall benefits of the program are computed 
and compared against the program's total cost, measured as the resources spent. The 
program cost measures are the performance standard for the program under this 
approach. Many workforce development projects, such as One Stop Centers and several 
states — including California and Oregon — plan to use this approach by computing the 
total benefit from reduced welfare cost and increased taxes paid by participants, and 
comparing it to the total cost of workforce programs. 

• Continuous improvement. A program's own past performance on measures, or 
performance according to some preset goal, is the standard upon which the program is 
judged, under the continuous improvement model. Since past performance is used to 
compute levels of acceptable performance, different standards are used to judge each 
local program. 

• External standards. Uniform standards are established for all programs, according to 
some empirical or policy-based criteria. These standards can be set to define minimal 
acceptable performance or exemplary performance. The current JTPA performance 
standards are the best example of this model. 

• Relative ranking. In this model programs are ranked according to their performance on 
measures and those scoring at the top are considered “successful,” while those at the 
lower ends of the distribution are “failures,” or at least in need of improvement. 

One difficulty adult education has experienced in trying to demonstrate its effectiveness is 
partly the result of the different assessment strategies used by stakeholders to evaluate the program. 
Adult educators have preferred the continuous improvement strategy of assessing their programs. 

Most states judge their local programs according to their progress toward meeting state goals for 
adult education, program-developed goals or past performance. The relative ranking model also is 
popular with adult educators. Other audiences outside the program, however, such as legislators, 
have judged the program according to its return on investment and external standards (i.e., retention 
is not long enough, participants do not leam enough). The problems for the program resulting from 
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these divergent views underscore the importance of establishing a performance assessment strategy 
that meets program needs. 

Issue 6: How can unintended effects of performance measurement be minimized? 

Performance measures are intended to allow stakeholders to assess whether a program is 
achieving its policy goals. Ideally, measures do not affect services and operational decisions of 
providers. Yet, when people know they are going to be evaluated on a set of criteria, they tend to 
change their behavior to meet these criteria. Rather than being a passive gauge of program 
performance, the measures become a determinant of program performance. This phenomenon is 
known as an unintended effect of performance measurement. 

Unintended effects are well known to anyone who has been involved in evaluating programs 
or individuals. Teachers teach to the test, for example, if they know they will be evaluated 
according to how their students perform on the test. Corporations focus upon short-term fiscal gain, 
if they know their stockholders will judge them by quarterly profits. It is only recently, however, 
that the impact of unintended effects upon performance management seriously have been studied. 

The Department of Labor has been a leader in this area in its study of how JTPA 
performance standards have affected service delivery (National Commission for Employment 
Policy, 1988; Greene, 1991, cited in Condelli and Kutner, 1992). The two most well-known 
unintended effects for JTPA were “creaming” of participants, where better prepared participants 
were given preference into the program over those more in need, and the development of local 
program service design that emphasized short-term, less expensive services. This latter effect was 
due to the cost per entered employment performance standard, by which programs were assessed 
according to the average cost to place a participant in a job. Due to its unintended effects, the 
Department of Labor eliminated the cost standard in 1991. 

While it is probably not possible to eliminate unintended effects completely, the JTPA 
research identified the following four ways to minimize them. Adult educators can build upon this 
information to avoid unintended effects as they develop their reporting system. 

• Understand how measures relate to policy goals. Performance measures should match 
the program’s policy goals and all stakeholders should be aware of the relationship 
between the measures and goals. In other words, local program providers need to know 
the purpose of the measures and how they will be used to assess performance. 

• Set performance objectives appropriately. A natural tendency often is to set 
performance goals high in the belief that better performance will result. We are often 
exhorted, for example, to “reach for the stars” or to develop “world-class” schools. The 
JTPA experience indicated, however, that no matter how high standards were set, 
programs were able to meet them — but they often did so by cutting services or costs, or 
“creaming” participants. Performance goals need to be set at a moderate level — enough 
to achieve policy objectives, but not so high as to trigger undesirable effects. Baseline 
data of typical program performance are needed to identify this ideal level of 
performance. 
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• Provide appropriate incentives and avoid sanctions. Just as performance objectives 
should be set at a moderate level, incentives for good performance should not be set too 
high. The more there is at stake for meeting a level of performance, the more programs 
will work to meet it and the more likely unintended effects become. To use an extreme 
example, if local adult education programs were funded primarily on the number of 
GEDs their students achieved, programs would reorient their instruction to achieve this 
goal. The result would be an absence of other types of students in adult education (e.g., 
low-literate and ESL students) and the instruction provided would not match other policy 
goals. The JTPA studies recommend avoiding sanctions altogether, as they usually bring 
unintended effects. In any case, it is so difficult to sanction a program financially that no 
JTPA program has been sanctioned due to poor performance (Department of Labor, 
1996). 

• Train and monitor local program staff. Training staff in the purpose and the use of data 
will minimize misunderstanding and misuse. A frequent shortcoming of current 
reporting systems, including adult education reporting, is that local staff do not know 
why and how states and the Federal government are using the data they collect (Condelli, 
1994). Monitoring local programs and obtaining input from local staff on the impact of 
performance systems, provides an opportunity to discuss the purpose of measures and to 
identify misunderstandings that can lead to unintended effects. 

Issue 7: What infrastructure and staff development are needed to operate the system? 

After measures, methodology and a performance standard strategy have been decided, the 
reporting system is ready to be implemented. Collecting and reporting of data requires a data 
collection infrastructure that includes precise, quantitative definitions of the measures, protocols or 
forms to collect the measures, an individual student record management information system (MIS) 
for local programs to collect the information, automated methods for aggregating data at the state 
and Federal levels, and a systematic method of staff development. 

Reporting System Infrastructure 

Although the exact flow of data and procedures will depend upon the type of system that is 
ultimately adopted, it is expected that local instructional sites will collect student data using a 
uniform MIS and report it to their program office. Each program will report these data to their state 
offices, and states, in turn, will report their data to the Federal office for aggregation and analysis. 

As noted in Chapter 1 , an individual student-based MIS is critical to the successful 
implementation of the reporting system. While a national student MIS could be developed for the 
reporting system, this is probably not advisable, given the diversity of computer hardware and 
software in use within states, as well as other data and reporting needs of local programs and states. 

A more workable approach is to develop a uniform set of definitions and codes for each measure and 
technical standards for the electronic form of the measures. States and programs could then use any 
software, as long as their system conforms to the technical standards. States also need software for 
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aggregating and combining data from individual sites and programs and at the Federal level, the 
Department of Education will need aggregation software for combining data from individual states. 

Although attempts are underway to improve the current state of the adult education reporting 
infrastructure, it is currently chaotic. A review of state data systems found a wide range of systems 
and procedures in use, including manual reporting systems, desktop computers and sophisticated 
statewide systems (Kutner, Webb and Herman, 1993). While there have been some improvements 
over the last five years within several states, aggregation of data at the state and Federal level is still 
done primarily through paper forms, even though automated reporting software, developed by 
DAEL, is readily available. Of the 35 states that responded to a background survey conducted for 
this paper, 24 collected data for the DAEL Annual Statistical Report in paper form from at least 
some of their local programs. Only six states reported data to DAEL in electronic form in the last 
program year. 

Staff Development 

Another key component to implementation of the reporting system is comprehensive staff 
development for state and local staff. While “staff development” is probably not the first thing that 
enters the mind when thinking about a reporting system, a moment’s reflection reveals it as essential 
to the system’s success. Since it is local program staff that collects and reports the information for 
the system, they must understand fully the procedures for collecting the data. Otherwise, data will 
be incomplete and invalid. Comprehensive, ongoing training is essential to successful 
implementation. 

Currently, few states devote significant resources to training local staff on data collection 
and reporting. This lack of training is a serious oversight that must be addressed. As has been 
emphasized above, local staff must understand the reporting system and its measures and how they 
relate to policy goals. Staff also must understand the definitions of measures, how to complete 
forms and protocols that collect the measures, and procedures for submitting and editing 
information. A program of staff development for local staff should consist of at least the following 
four components to ensure high-quality data for a reporting system. 

• Involve local staff in data system development. At the outset, local staff, including 
teachers, should be involved in developmental activities. Their input should be included 
in decisions about the system’s basic components, such as reporting procedures and 
frequency of data collection. 

• Initial training on the data system. This training should be attended by local staff with 
the actual responsibility for entering and managing the data. It should include training 
on the basic components of the reporting system, such as definitions, use of forms and 
reporting procedures, as well as training on the program’s management information 
system. 

• Follow-up training. This training should be held several months after the initial training 
and be attended by the same local staff who attend the initial training. The training, held 
after staff has had some time using the system, providesran opportunity to answer 
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questions, discuss problems and learn more detailed information about its purpose and 
use. 

• Data organization. It is relatively easy to leam definitions, how to use forms and how to 
use software. The bigger challenge to most local staff is how to organize forms and data 
procedures to make record keeping efficient and easy to perform. For example, 
procedures for who completes forms, who keys them into the computer, who is 
responsible for keeping data up-to-date and who maintains the system, need to be 
resolved. Training on these topics is often overlooked, but is essential to getting good, 
timely data. 

Local programs also need ongoing support and monitoring to reinforce training. One or 
more state data coordinators can fill this role by being available to answer questions from local staff 
as they run into difficulties. The state data coordinator also can monitor local program activities to 
ensure compliance and to provide quality control throughout the year. 

Table 9 presents a summary of these main infrastructure needs for an adult education 
outcome reporting system. 



TABLE 9 

Main Infrastructure Components and Implementation Requirements 
for Adult Education Reporting System 



'• CpMPdNENT> 


. r REQUIREMENTS ' ^ * 


Measures and Methods 


• Standard definition 

• Standard methodology 

• Uniform protocols or forms 


Management Information System 


• Individual student-based records 

• Uniform technical formats 

• Ability to aggregate data from local sites 
to program-level and from program-level 
to state-level 


Staff Development 


• Initial and ongoing training 

• Based upon input from teachers and 
local staff 

• Data organization training 
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CHAPTER 4 




Measures and Methods 


One of the major tasks in developing a national outcome reporting system is to identify 
candidate measures that reflect adult education’s policy goals. To assist with this task, this chapter 
presents the following eight sample measures for each of the outcome areas of the Columbia 
Framework. 


• Learning Gains 


- Attainment of academic or functional skill 


• Economic Impact 


- Improved employment situation 

— Reduction in receipt of public assistance 


• Credentials 


- Attainment of an educational or skill credential, 

including GED 


• Further Education 
and Training 


- Entry into another education or training program 


• Family Impact 


- Increase in literacy-related or school activities with 

children 


• Community Impact 


- Increased involvement in community 


• Customer Satisfaction 


- Expressed satisfaction with instructor and program 

services 



These are the measures most commonly used by planned and existing reporting systems, as 
identified through the review presented in Chapter 2. Of course, the reporting system needs not be 
limited to these measures, and additional measures may be identified as the system develops. 

The presentation in this chapter is designed to assist in consideration of the measures by 
revealing the major issues that need to be addressed in their implementation. A sample definition is 
offered for each measure, along with possible sources for the data, a methodology, a brief overview 
of relevant methodological issues and a listing of current systems that use the measure. For some 
measures, up to three methodological options, corresponding to the methods most commonly used 
by existing reporting systems, are presented. Chapter 2 presents more detail about current usage and 
the methods issues related to the measures, and Issue 5 in Chapter 3 briefly discusses the three 
methodologies and technical issues. Chapter 5 offers further discussion of possible reporting system 
methodologies. 
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Learning Gains 


Measure and Definition 


Attainment of academic or functional skills — number of 
students who obtain skills associated with an education level. 


Sources 


Direct program report based upon pre- and post-assessment of 
student. 


Methodology 


Functioning levels would need to be defined with specific skills 
and competencies associated with each level (e.g., DAEL 
educational functioning level definitions listed in Appendix or 
CASAS levels). Standard assessment benchmarks would be tied 
to each level and program staff would assess students at intake 
and after an appropriate number of instructional hours, using the 
standard assessment method. Programs would report the 
number of students who advance one or more levels. 


Data Collection Issues 


Requires observable and clearly specified skills associated with 
each level, a uniform method of assessment among local 
programs and training of local program staff on these definitions 
and assessment. 




Post-testing is difficult, since many adult education students 
leave before they can be tested. 




Different test and skill levels used by local programs would 
have to be translated into a single, national standard. For 
example, tests would have to be linked and skill level 
equivalencies defined. 


Current Usage 


This measure is the main outcome measure of DAEL’ s reporting 
system and is used in some form in all states. The new (i.e., 

1 996-97) DAEL reporting forms have specific definitions and 
test benchmarks for levels (see Appendix). 




Several states have developed functional levels for reporting, 
including California, Oregon, Connecticut, Iowa, Texas, 
Tennessee and Kentucky. 




Twelve states use the CASAS framework to define levels. 




JTPA-funded programs have defined skill levels for youth 
employability enhancement, which are defined and assessed 
locally. 
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NIFL’s EFF project and programs in several states are working 
to develop competencies that can be translated to definable skills 
for assessment. 

NCFL’s reporting system includes TABE, GED and ESL test 
scores. 

One Stop Centers will collect attainment of workplace skills, 
defined locally or by state. 
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Economic Impact 


Measure and Definition 


Improved employment situation — (1) number of learners 
who obtain a job, (2) remain in their current job, or (3) are 
promoted to a job that requires higher skills as a result of 
adult education instruction. 


Sources 


Self-report of learner, possibly verified by employer; direct 
program placement and reporting; other program data 
systems. 


Methodology 


Local staff collects learner’s employment status at intake and 
again at some time shortly following participation. The 
measure could be collected either of all students or a sample 
of students, using any of the following methodologies. 




Direct program placement and reporting. The program 
assists in or places the learner in a job, or has direct 
knowledge of changed employment status, and reports 
placements and other employment outcomes. 




Follow-up survey. The program conducts a phone or mail 
follow-up survey to determine employment status. To 
improve validity, the information from learners could be 
verified by contacting a sample of employers. 




Data linking. Adult education student data are linked with 
employment data bases (e.g., unemployment insurance or 
Social Security records) to identify learner employment 
history. 


Data Collection Issues 


Employment goals may not be appropriate for all learners or 
all programs. The validity of the measure would be 
improved, if used only for learners with employment goals or 
if used in adult education programs with instruction related to 
employment. 




Except in adult education programs specifically designed to 
teach workforce skills, the relationship between program 
activities and employment is indirect and may take 
considerable time to emerge (see Chapter 3, Issue 3). 




Follow-up surveys are costly, time-consuming and require 
sampling capabilities. 
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Data linking requires a common, shared interagency data base 
with common learner identifier. Requires interagency data 
sharing arrangements. 

Current Usage DAEL Annual Statistical Report and the NCFL Reporting 

System ask programs to report the number of learners who 
obtained employment. The DAEL Report also asks for 
number of learners who secured employment retention or 
promotion. 

JTPA programs collect employment information on adult 
learners 13 weeks after training ends, and record youth 
employment, using follow-up methodology. 

One Stop Centers, PMRIS states and NGA states will collect 
entered employment, job placement rate and job retention. 

Oregon, Kentucky and California plan data-sharing 
arrangements to measure net economic gain to state based 
upon learner employment. 

Florida and Illinois link interagency data to compute this 
measure. 
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Economic Impact 



Measure and Definition 


Reduction in receipt of public assistance — number of 
learners who are removed from public assistance as a result of 
adult education instruction. 


Sources 


Self-report of learner, possibly verified by welfare records; 
direct program report; other program data systems. 


Methodology 


Local staff records whether learner is receiving public 
assistance at intake and again at some time shortly following 
participation. The measure could be collected either of all 
students or a sample of students, using any of the following 
methodologies. 

Direct program reporting. The program has direct 
knowledge of learners’ public assistance status, and reports 
removal from assistance while learner receives instruction or 
shortly thereafter. 

Follow-up survey. The program conducts a phone or mail 
follow-up survey of learners to determine public assistance 
status. To improve validity, the information from learners 
could be verified by contacting welfare offices. 

Data linking. Adult education student data are linked with 
public assistance record data bases to determine which 
learners are receiving benefits. 


Data Collection Issues 


Since this measure only is relevant to learners receiving 
assistance, it should only be applied to them or to programs 
designed specifically to serve them, such as programs for 
welfare-work learners. 

Except in adult education programs specifically designed to 
address the needs of public assistance recipients, the 
relationship between program activities and reduction in 
welfare is indirect and may take considerable time to emerge 
(see Chapter 3, Issue 3). 

Follow-up surveys are costly, time-consuming and require 
sampling capabilities. 
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Data linking requires a common, shared interagency data base 
with common learner identifier. Requires interagency data 
sharing arrangements. 

Current Usage DAEL Annual Statistical Report asks programs to report the 

number of learners who are removed from public assistance. 

JTPA programs collect employment status of welfare 
recipients 13 weeks after training ends, using follow-up 
surveys. 



One Stop Centers, California, Illinois, Kentucky (PMRIS), 
Tennessee (PMRIS), Oregon and NGA states plan to look at 
welfare caseload reduction through interagency data sharing. 

California plans to measure employment rate for welfare 
recipients through interagency data sharing. 
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Credentials 


Measure and Definition 


Attainment of an educational or skill credential — number 
of learners who attain a GED credential or other formally 
recognized credential that has associated academic or 
workplace skills. 


Sources 


Direct program report, based upon testing of learner. 


Methodology 


States would develop certificates of competency that certify 
learner has attained specific skills. GED test areas also 
would define credentials. Program tests to measure learner 
attainment of specific skills would tie to credentials or to 
GED test areas, and would record the number of students that 
pass tests and achieve the credential. If the program does not 
administer the test, the testing center submits the information 
on learners to the program for reporting. 


Data Collection Issues 


Except for already existing certification, such as GED, a set 
of meaningful skills need to be defined by states or DAEL for 
certification levels to be valid. If states define levels, a 
Federal framework would need to be developed to ensure 
consistency. 




Programs that do not administer the certification test directly 
must develop a reliable system for obtaining test results from 
the agency or center that does the testing. 




It can take several years for learners to achieve the skills 
related to the credential, so that program impact may not be 
readily apparent. 


Current Usage 


All existing, planned and demonstration reporting systems 
related to adult education collect this measure. 




Colorado, Delaware and Oregon have formal certification 
programs tied to competencies. 
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Further Education and Training 



Measure and Definition 


Entry into another education or training program — 

number of learners who enter a higher education program, 
vocational education or a job training program following 
adult education instruction. 


Sources 


Self-report of learner, possibly verified by other education or 
training program; direct program report; other program data 
systems. 


Methodology 


This measure could be collected either of all students or a 
sample of students, using any of the following methodologies. 

Direct program reporting. Local program directly places and 
reports learner entry into another program or receives this 
information directly from other programs for reporting. 

Follow-up survey. The program conducts a phone or mail 
follow-up survey of learners to determine entry into other 
education or training. To improve validity, the information 
from learners could be verified by contacting other education 
and training programs. 

Data linking. Adult education student data are linked with 
data bases of other education and training programs and 
community colleges to determine whether students enrolled in 
them. 


Data Collection Issues 


Follow-up surveys are costly, time-consuming and require 
sampling capabilities. 

Data linking requires a common, shared interagency data base 
with common learner identifier. Requires interagency data 
sharing arrangements. 

All other education and training programs may not have 
students record systems to link with adult education, meaning 
some learners’ advancement would not be identified, if the 
data sharing method is used. 


Current Usage 


DAEL Annual Statistical Report asks programs to report the 
number of learners who entered other academic or vocational 
programs. 
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As part of youth employability enhancement, JTPA programs 
report number of youth who entered non- JTPA training and 
stayed for at least 90 days or 200 hours. 

The NCFL reporting system counts whether learner entered 
community college, university, vocational or job training 
programs. 

NGA states record number of learners who return to school or 
enter post-secondary training. 

Kentucky PMRIS project defined this measure as the 
percentage of participants who enter other education or 
training, or the military, within 12 months after receiving a 
diploma. 

California plans to measure the number of participants who 
enter public, post-secondary education using data sharing. 
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Family Impact 



Measures and Definition Increase in literacy-related or school activities with 

children — (1) number of times per month learners perform 
literacy activities with their children, such as reading to 
children and helping children with school work; and (2) 
number of times learners help children with school work, talk 
to children about school, talk to children’s teachers or 
participate in activities at children’s school. 

Sources Direct program report based upon self-report of learner. 

Methodology This measure could be collected through a survey of learners, 

administered at intake and then periodically (e.g., monthly) 
while the learner attends the program. Alternatively, a 
follow-up survey on a sample of learners could collect the 
measures after learners leave the program. 

Data Collection Issues This measures relies on self-report of learner, which has 

notorious validity problems due to social desirability issues 
and unreliability of memory. 

The measure is only relevant to family literacy programs or 
learners with a family literacy goal. Otherwise, the 
relationship between program activities and family literacy 
activities is indirect and may take time to emerge (see 
Chapter 3, Issue 3). 

Follow-up surveys are costly, time-consuming and require 
sampling capabilities. 

Current Usage Beginning in Program Year 1996, DAEL Annual Statistical 

Report asks family literacy programs to report the number of 
learners who read more to their children and have greater 
involvement in their children’s school. 

NCFL reporting system includes a parent survey, 
administered weekly, that asks parents to record these 
measures. 

Hawaii and Kentucky PMRIS projects planned to measure 
number of parents who volunteer in school, read to children, 
or participate in family community network. 
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Community Impact 


Measures and Definition 


Increased involvement in community - (1) number of 
learners who achieve citizenship, (2) number of learners who 
register to vote or vote for the first time, and (3) increase in 
involvement in community activities, such as volunteering in 
community organizations, attendance at neighborhood 
meetings and community activism. 


Sources 


Direct program report based upon self-report of learner. 


Methodology 


This measure could be collected through a survey of learners, 
administered at intake and then periodically (e.g., monthly) 
while the learner attends the program. Alternatively, a 
follow-up survey on a sample of learners could collect the 
measures after learners leave the program. The survey would 
collect citizenship status, prior voting behavior, and measures 
of community participation. 


Data Collection Issues 


This measure relies on self-report of learner, which has 
notorious validity problems due to social desirability issues 
and unreliability of memory. 

The measures may not be relevant to all learners, and would 
apply most directly to programs with community education 
goals. Citizenship would apply primarily to ESL students. 

Except in programs with community education goals, the 
relationship between program activities and community 
activities is indirect and may not emerge (see Chapter 3, Issue 
3). 

Follow-up surveys are costly, time-consuming and require 
sampling capabilities. 


Current Usage 


The DAEL Annual Statistical Report asks programs to report 
the number of learners who achieved U.S. citizenship and 
who registered to vote or voted for the first time. 

PMRIS projects planned to measure voting (Hawaii), 
community involvement (Kentucky) or citizenship attainment 
(Tennessee). 
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Customer Satisfaction 



Measures and Definition Expressed satisfaction with instruction and program 

services by learners, employers and other education and 
training providers. 

Sources Direct program report based upon self-report of learners, 

employers and providers of other education and training 
programs. 

Methodology This measure could be collected through a follow-up survey, 

administered several months after learners leave the program. 
The survey would include samples of learners, employers 
who directly hire adult education program participants, and 
staff of other programs that enroll adult education learners. 
For learners, the survey would measure their satisfaction with 
instruction and services. For employers and other program 
providers, the survey would assess whether they were 
satisfied with the incoming skills of the learners who 
participated in adult education. 

Data Collection Issues Surveys of satisfaction would have to be developed and tested 

for validity and reliability. 

Learners will have to perceive the survey as independent from 
the program and not tied to receipt of instruction, to avoid 
pressure for socially desirable response. 

This measure is appropriate for program improvement 
purposes, but does not address goals of the program. 

Follow-up surveys are costly, time-consuming and require 
sampling capabilities. 

Current Usage One Stop Centers, California and Oregon plan to survey 

employer satisfaction. 

Oregon plans to measure participant satisfaction through 
surveys. 
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CHAPTER 5 

Where We Might Go: Four Models 

In this chapter we put together the key issues, measures and lessons learned from other 
programs and present four sample national outcome reporting systems that can be considered for 
adult education. These sample systems reflect the dominant approaches to program accountability 
reporting and are offered to facilitate thought and discussion as the development of a national 
reporting system proceeds. This presentation is not intended as an endorsement of any approach and 
other models are likely to be developed over time. The models, however, represent a wide range of 
possibilities and demonstrate how many of the issues and measures discussed in this document can 
be resolved to develop a cohesive, comprehensive reporting system. All of the models assume that 
local programs would have automated, individualized student record systems that include uniformly 
defined measures for reporting. 

• Model 1: Comprehensive Reporting Model is an enhancement of the current DAEL 
reporting system and relies on local programs to collect a broad and comprehensive set 
of measures. 

• Model 2: Follow-up Survey Approach also would collect a broad set of measures, but 
includes a statewide follow-up sample survey. 

• Model 3: Data Sharing/Workforce Model relies extensively on interagency data sharing 
and reflects the model currently under consideration by workforce development projects, 
such as One Stop Centers and by California, Florida, Illinois, Kentucky and Oregon. 

• Model 4: Menu Approach would allow programs to select the measures that apply to 
them from a common core of standard measures. Some workforce development projects 
have considered this model. 

We describe the measures and methods of each option below, along with an assessment of its 
advantages and difficulties that could be encountered in its implementation. 

Model 1 : Comprehensive Reporting Model 

This option enhances the existing DAEL reporting system by standardizing methodology and 
measures, and by including procedures to ensure more uniformity in implementation and oversight 
of the data collection process. Local programs would collect a broad range of measures, as they do 
now, and report them to their state education agency for reporting to DAEL. 

Measures and Methods 

The current reporting system already collects all of the measures of the Columbia 
Framework, except customer satisfaction, and local programs would continue to collect these 
measures, as currently defined (see Chapter 2). To include all of the Columbia Framework 
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measures, a new measure of satisfaction of learners within the program would be added. Local 
programs would collect measures of satisfaction, through a questionnaire, when learners leave the 
program or at the end of the program year. 

Implementation of the methodology for this option would focus upon improving the two 
major shortcomings of the system as it now exists: the lack of uniform definitions, especially for 
educational functioning level; and the incomplete reporting of the economic, family and community 
impact measures that require follow-up. DAEL already has begun work toward improving 
definitions of educational levels with the publication this year of specific reading, writing and 
functional skills (as well as speaking and listening skills for ESL learners) tied to each level. DAEL 
also suggested test benchmarks for each level (see Appendix). 

The first activity in implementation of this option would be to ensure a uniform adoption of 
these definitions within the states. State staff and practitioners would work with DAEL to refine, if 
necessary, the educational level definitions and assessment benchmarks. DAEL also would work 
with states that have their own level definitions and assessment methods to develop crosswalks that 
translate the state definitions and assessments to correspond to the Federal standards. DAEL would 
offer technical assistance to states to ensure uniform implementation of the definitions by local 
programs. A system of quality control, through monitoring of local data collection activities, also 
would be implemented in each state to ensure compliance and to identify program staff in need of 
further training. 

Currently, there are no Federal guidelines for collecting the economic, family and 
community measures. Consequently, a second activity needed to implement this option would be for 
DAEL to develop a protocol and methodological guidance for states and local programs in collecting 
these measures. For example, a protocol would include the following information. 

• Definition and identification of applicable learner population. As the measures in these 
areas do not apply to all learners, programs only would need to collect them from learners 
where applicable (e.g. welfare reduction measures from welfare clients, family measures 
from family literacy programs). Definitions of each population group for each measure 
would be provided. 

• When to collect the measure. Guidance on when to collect the measures would be 
developed. Measures can be collected during participation (i.e., monthly), at the end of the 
year, when a student leaves or at some time after the learner leaves. 

• Sampling method. Program staff would collect the measures either of all or a sample of 
students. A simple, easy-to-implement sampling methodology would be provided for 
programs that choose to sample. 

• Data definitions or questions. The protocol would include actual questions or define the 
data element to measure. For example, questions to elicit whether the learner "reads more to 
children" would be included. 
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• Recording of missing data. Local programs need to report the number of non-responses, or 
learners they are unable to locate, when collecting these data, to allow for assessment of their 
validity. 

DAEL would support training and technical assistance to state and local staff on use of the protocol 
to ensure uniform implementation. States also would need to develop quality control procedures, 
including onsite monitoring and “spot checking,” to verify local implementation. 

Assessment of the Model 

As this option builds upon what is already in place, it can be implemented with the least 
disruption and change to the delivery system. The wide range of measures reflect the broad goals of 
the adult education program and, therefore, can be used to evaluate the program in several areas and 
for diverse audiences. If implemented properly, the standardization of measures and methods, and 
technical assistance would greatly improve the quality of data that would be produced by this 
system. 



A problem with the approach is that it relies exclusively upon local program staff to collect 
and to report a wide range of learner data. It is unclear whether programs have the resources and 
ability to collect all measures with a high level of quality, especially the economic, family and 
community measures. Other problems with these latter measures are that the validity of some of 
them may be problematic (e.g., reading to children) and that performance on some measures does 
not directly result from program instruction (e.g., welfare reduction cannot be directly linked to 
instruction — see Chapter 3, Issue 3). 

Model 2: Follow-up Survey Approach 

The burden of participant follow-up for local programs is lessened or removed under this 
approach, since local programs would collect only the educational measures on learners under this 
model. A follow-up survey, conducted by the state, would collect economic, family, community and 
satisfaction measures. Local programs would not be required to do participant follow-up. 

Measures and Methods 

Local programs would be responsible for assessing and reporting measures of student 
learning gains for all learners, and, where they have direct knowledge, learner attainment of 
credentials and advancement to other education and training. The implementation of these measures 
would be the same as for Model 1 . DAEL's uniform definitions of skill levels and assessment 
methods would have to be implemented, along with appropriate crosswalks to accommodate state- 
developed skill levels and assessment methods. Training and technical assistance, as well as quality 
control checks by the state on local assessment procedures, would be required. 

Unlike Model 1 , however, local programs would not collect follow-up data from 
participants. An annual, statewide, follow-up telephone survey would collect these economic, 
family, community and satisfaction measures. The state education agency would be responsible for 
conducting the survey by drawing a sample of learners who have attended and left adult education 
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during the last program year. The sample would be drawn from student data submitted to the state 
by local programs. A two-stage sampling frame would be employed, where programs are first 
selected, in proportion to their size; students within programs then would be selected randomly. 

To ensure uniformity of measures and methods, DAEL would develop sampling guidelines, 
data collection procedures and uniform data elements to collect the information. DAEL also would 
provide technical assistance and training to state staff in conducting the telephone survey. 

Assessment of the Model 

Like Model 1, this system collects a broad set of measures to allow evaluation of the 
program in several areas for multiple audiences. In addition, higher data quality is likely for the 
follow-up measures, through the centralized follow-up survey. Potential problems due to lack of 
resources and use of uniform data collection procedures by local programs are eliminated. Since 
only 50 states, as opposed to several thousand local programs, would conduct the follow-up survey, 
it would be easier to maintain consistency and control over measurement through technical 
assistance and training. 

The success of this option depends upon proper sampling methods and the ability of states to 
implement the survey. To conduct good sampling, states would need accurate and timely 
information about local student enrollments. Programs would have to be able to submit identifying 
and demographic data on individual students to the state, as well as accurate breakdowns of 
enrollment by student type and program, to allow a scientifically valid sampling frame that would 
conform to Federal guidelines. 

States would need the resources to conduct the survey quickly, either directly or through an 
outside contract. A major difficulty in conducting the survey is the transient nature of the adult 
education population, which may make it difficult to locate learners. In addition, some learners may 
not have a telephone, further complicating the logistics of conducting the survey. 

Finally, the ability of a follow-up survey to identify improvements on these measures 
resulting from adult education is not clear and attempts to do so have met with little success (Cohen, 
Condelli and Garet, 1996; Development Associates, 1994). The lack of success may be due to the 
fact that program instruction does not directly influence some measures (e.g., except in a workplace 
program, instruction is not directly linked to employment) and consequently may take considerable 
time and a confluence of other variables (e.g., the local economy, learner motivation) to emerge (see 
Chapter 3, Issue 3). In addition, social desirability of responses for learners is a threat to validity. 

For example, learners may overreport attendance in other programs, because they know this is a 
desirable outcome. 

Model 3: Data Sharing/Workforce Model 

Several One Stop Center states, PMRIS states, and other states developing performance- 
based workforce accountability projects, such as California, Florida, Kentucky and Oregon, are 
considering this approach to accountability. In this model, all adult education and training programs 
are viewed as components of a single workforce development system. Each program reports only 
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measures directly relevant to its purpose, and data from each system are pooled into a shared data 
system for analysis of short- and long-term outcomes. 

Methods and Measures 

Adult education programs would report only educational measures — learning gains and 
credentials — along with student identifying information, to the state education agency. These data 
would then be combined with data from other education and training programs of the state, 
unemployment insurance records, welfare records and Social Security data bases, into a shared 
system for analyses of economic impact measures (e.g., employment rate, job retention, wages, 
removal from public assistance) and enrollment in further education and training programs. The 
state also would conduct a customer satisfaction survey of a sample of employers and learners. 

As with other models, adult education programs in each state would need to standardize their 
systems of assessment and definitions of educational functioning level to implement this system. 
However, unlike the other models, programs would have to report individual student record data 
with a common identifier, such as Social Security number, to the central state data system. In 
addition, each state would need to develop a data sharing agreement among agencies that would 
specify: 

• The technical format for all agencies to submit data; 

• Common definitions for data elements that all agencies would use; 

• The agency responsible for collecting and maintaining the data; 

• The purposes for which the data could be used; and 

• How confidentiality would be ensured. 

Data analysis software also would need to be developed that would allow for combining and 
analyzing the data. 

Assessment of the Model 

This model explicitly narrows the scope of adult education, defining it as an education 
program and one component of an employment-based system. The advantages of this approach are 
that adult education is directly accountable only for outcomes it most directly affects (i.e., teaching 
of functional and academic skills), and it minimizes the data collection and reporting burden on local 
programs by focusing only on educational outcomes. 

A major disadvantage, however, is that interagency data sharing arrangements are extremely 
difficult to implement. Welfare, education, labor and social service agencies must collaborate in 
each state to establish common definitions for measures, data transfer technical standards, and a 
shared data system and software for the combined data. Each agency must have an individual record 
data system that includes a common identifier, typically Social Security numbers, which many 
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programs are reluctant to collect due to concern over confidentiality issues. Further, many local 
adult education programs currently lack the capacity to transmit individual student record data 
electronically. 

Control of the data also is a major issue, since one agency must take the lead in collecting 
and combining the data from other agencies. Other agencies may be reluctant to give up control of 
their data to another agency, which can then analyze and report it in ways other than originally 
intended by the source agency. In some states, there also are legal prohibitions against the release or 
collection of some data, which makes data sharing difficult. For example, some state agencies 
cannot collect or share clients’ Social Security numbers, wage or employment data; some 
practitioners also raise ethical issues against the use of such data. 

There are further difficulties with this model. It would likely require a Federal mandate 
before states would act, given the degree of disruption it would cause within several agencies in each 
state. This model also could not be implemented quickly, since it requires substantial interagency 
collaboration and change. Connecticut and Oregon, two states that have considered this approach, 
report that it takes years to develop and implement. Data sharing is not yet in operation in either 
state, despite years of planning. The procedures took several years to develop in Florida and Illinois, 
states that currently implement data sharing (see Chapter 2). 

Model 4: Menu Approach 

The “menu” in this model is a list of outcome measures with standard definitions and 
methodologies for collecting them. All local programs would be required to use the learning gain 
measures but, in addition, would select from the menu any other outcome measures relevant to them. 
Programs then would be held accountable on these measures only. The Department of Labor is 
planning to promote this type of approach for employment programs through its Workforce 
Development Performance Measures Initiative. 

Methods and Measures 

As with other models, the Department of Education would establish standard definitions and 
methodologies for each of the measures of the Columbia Framework, along with technical reporting 
requirements. To ensure some consistency of measures across programs, all programs would be 
required to use a core set of measures that would include at least the learning gains measure. 
However, local programs also would select any additional measures that most reflect their program 
goals, subject to state approval. Local programs would be required to use the standard definitions 
and methodologies and report these outcomes annually. Each state then would report outcomes from 
all of their programs to DAEL, along with information on the number and type of programs that used 
each measure, and the number of students to which the measures were applied. 

Assessment of the Model 



This model clearly defines adult education as an education program by requiring core 
measures of learning gains, while also recognizing the multiple goals of adult education by offering 
other outcome options that states can identify. The main strength of the approach is that programs 
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are held accountable only for the outcomes they explicitly try to affect. For example, workplace 
programs could focus upon employment outcomes, and family literacy programs would be assessed 
on the family impact measures. 

While this option would work well in local programs that have only one or two student 
outcome goals, most programs have multiple instructional goals. For example, a program may have 
GED, family and workplace components, requiring it to use measures in each of these areas. As a 
result, the reporting for most programs would end up being very much like current reporting or like 
Model 1. 

The multiple goals of most programs also would make this model very difficult to implement 
at the local level. Individual students within the same site or program would have to be assessed on 
different outcomes. For example, if a site offered family literacy classes, general ABE classes and 
ESL instruction, the site would have to report family impact outcomes for family students, ESL 
outcomes (e.g., citizenship) for ESL students, and learning gains for all students. 
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Appendix 



DAEL EDUCATIONAL FUNCTIONING LEVEL DEFINITIONS 



Adult Basic Education Level 

Beginning Literacy 

■ Functional skills. The individual cannot read basic signs or maps, cannot complete 
simple forms and has few or no basic computational skills. 

■ Reading and writing. The individual has no or minimal literacy skills and little or no 
formal schooling. There is little or no recognition of the alphabet. 

■ Computation. The individual has little computational skills and has little 
recognition of numbers. 

■ Test benchmark. Grade level 0-1.9; less than 200 CASAS. 

Beginning ABE 

■ Functional skills. The individual is able or could learn to read simple directions, 
signs and maps with some difficulty, fill out simple forms and perform simple 
computations. 

■ Reading and writing. The individual can read and print letters, one syllable words, 
most two syllable words and some three syllable words; can alphabetize words and 
write name, address and phone number; can write simple messages and letters 
using simple and compound sentences; can use simple punctuation (e.g., periods, 
commas, question marks). 

■ Computation. The individual can count, add and subtract three digit numbers, can 
perform multiplication through 12; can identify simple fractions. 

■ Test benchmark. Grade level 2-5.9; 200-210 CASAS. 

Intermediate ABE 

■ Functional skills. The individual is able to handle basic reading, writing and 
computational tasks; is also able or could learn to read simple employee handbooks, 
interpret a payroll stub, complete a basic job application or medical information 
forms, and reconcile a bank statement. May have difficulty with calculations, 
however, such as gas mileage. 

■ Reading and writing. The individual can paraphrase written narratives containing 
poly-syllabic words within all types of sentence structures; can compose a 
paragraph using combinations of all major sentence types; can review a paragraph 
and identify spelling and punctuation errors; can interpret actions required in 
specific written directions and write instructions to others, such as describing how 
to get to a specific address, or write a simple phone message. 

■ Computation. The individual can perform all four basic math operations with 
whole numbers and fractions; can solve narrative math problems requiring use of 
fractions and decimals. 

■ Test benchmark. Grade level 6-8.9; 211-235 CASAS. 
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Adult Secondary Education 

Individuals at this skill level should be attending an adult high school, an external 

diploma program or an ABE program with a goal of passing the GED tests. 

■ Functional skills. Individuals are able or could learn to follow simple multi-step 
directions, read common legal forms and manuals, write accident or incident 
reports, create and use tables and graphs, use math in business transactions and 
communicate personal opinions in written form. 

■ Reading and writing. The individual can comprehend expository writing and 
identify spelling, punctuation and grammatical errors; can find, summarize and use 
information from literacy works, magazines and professional journals to compose 
multi-paragraph essays on historical, contemporary and political issues; can listen 
to oral instructions and write an accurate synthesis of them; can work productively 
and collaboratively in groups. 

■ Computation. The individual can interpret and solve algebraic equations, tables and 
graphs and can develop own tables and graphs; understands and can apply 
principles of geometry to measure angles, lines and surfaces; can make 
mathematical estimates of time and space. 

■ Test benchmark. Grade level 9-12; over 235 CASAS. 

ENGLISH-AS-A-SECOND LANGUAGE LEVELS 

Beginning Literacy 

■ Functional skills. The individual functions minimally or not at all in English and can 
communicate only through gestures or a few isolated words. The individual may 
lack literacy in the native language and has had little or no formal schooling. 

■ Reading and writing. The individual cannot read or write or can read or write only 
isolated words. There may be little or no alphabet recognition. 

■ Speaking and listening. The individual cannot speak or understand English, or 
understands only isolated words or phrases. 

■ Test benchmark. 165-180 CASAS; SPL 0-1. 

Beginning ESL 

■ Functional skills. The individual functions with difficulty in situations related to 
immediate needs and in limited social situations; has some simple oral 
communication abilities using simple learned and often repeated phrases. 

■ Reading and writing. The individual has a limited understanding of print only 
through frequent re-reading; can copy words and phrases and write short 
sentences. 

■ Speaking and listening. The individual can understand frequently used words in 
context and very simple phrases spoken slowly and with some repetition; survival 
needs can be communicated simply, and there is some understanding of simple 
questions. 

■ Test benchmark. 181-200 CASAS; SPL 2-4. 
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Intermediate ESL 

■ Functional skills. The individual can meet basic survival and social needs, can follow 
some simple oral and written instruction and has some ability to understand on the 
telephone. 

■ Reading and writing. The individual can read simple material on familiar subjects, 
but has difficulty with authentic materials; can write simple paragraphs on survival 
topics and personal issues with some error. 

■ Speaking and listening. The individual can understand simple learned phrases and 
new phrases containing familiar vocabulary; can converse on familiar topics beyond 
survival needs; can clarify speech through rewording and asking questions. There 
is a use and understanding of basic grammar. 

■ Test benchmark. 201-220 CASAS; SPL 5-6. 

Advanced ESL 

■ Functional skills. The individual can understand general conversations, participate 
effectively in familiar situations, satisfy routine survival and social needs and follow 
oral and written instructions. Individuals also can understand conversation 
containing some unfamiliar vocabulary on many everyday subjects, but may need 
repetition, rewording or slower speech. 

■ Reading and writing. The individual can read materials on abstract topics and 
descriptions and narrations of factual material. The individual can write 
descriptions and short essays and can complete complex forms and applications. 
There is a general ability to use English effectively to meet most routine social and 
work situations. 

■ Speaking and listening. The individual can converse with no or minimal difficulty in 
conversation, can communicate over the telephone on familiar subjects and has 
basic control of grammar; understands descriptive and spoken narrative and can 
comprehend abstract concepts in familiar contexts. 

■ Test benchmark. 221 and above CASAS; SPL 7-10. 
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